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Abstract 


When algorithmic skeletons were first introduced by Cole in 
late 1980 dSOl the idea had an almost immediate success. The 
skeletal approach has been proved to be effective when ap¬ 
plication algorithms can be expressed in terms of skelefons 
composition. However, despife bofh fheir effectiveness and 
fhe progress made in skelefal sysfems design and implemen- 
fafion, algorithmic skeletons remain absent from mainsfream 
pracfice. Cole and ofher researchers, respectively in dSTTl and 
(ED, focused fhe problem. They recognized fhe issues affecf- 
ing skelefal sysfems and sfafed a sef of principles fhaf have 
fo be fackled in order fo make fhem more effective and fo 
fake skelefal programming info fhe parallel mainsfream. In 
fhis fhesis we propose fools and models for addressing some 
among fhe skelefal programming environmenfs issues. We 
describe fhree novel approaches aimed af enhancing skele¬ 
fons based sysfems from differenf angles. Firsf, we presenf 
a model we conceived fhaf allows algorifhmic skelefons cus- 
fomizafion exploifing fhe macro dafa-flow absfracfion. Then 
we presenf two resulfs abouf fhe exploifafion of mefaprogram- 
ming fechniques for fhe run-fime generafion and opfimiza- 
tion of macro dafa-flow graphs. In particular, we show how 
fo generafe and how fo optimize macro dafa-flow graphs ac¬ 
cordingly bofh fo programmers provided non-funcfional re- 
quiremenfs and fo execufion platform feafures. The lasf re- 
sulf we present are the Behavioural Skeletons, an approach 
aimed at addressing the limitations of skelefal programming 
environmenfs when used for fhe developmenf of componenf- 
based Grid applicafions. We validafed all fhe approaches 
conducting several fesf, performed exploifing a sef of fools 
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developed. 
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Chapter 1 

Introduction 


Computers are becoming tools of vital importance. They are used almost 
ever 5 rwhere, they are used for work, for study, for fun and actually for 
solve problem. Unfortunately, many problems require a huge amount of 
computational power to solve (as an example: genome mapping, portfo¬ 
lio risk-analysis, protein folding). Such a power cannot be obtained using 
a single processor. The only suitable solution is to distribute the applica¬ 
tion workload across many different computational resources. Resources 
those contemporaneously ("in parallel") execute parts of the whole ap¬ 
plication. Programming applications that make use of several computa¬ 
tional resources at the same time introduces some difficulties, as an ex¬ 
ample the communication and S5mchronization among the resources, or 
the application code and data decomposition and distribution. In order 
to ease this burden, since the early steps of computer science, researchers 
conceived and designed programming models and tools aiming at sup¬ 
porting the development of parallel applications. Throughout the ages, a 
lot of models and tools have been proposed, presented in several differ¬ 
ent (sometime exotic) forms. Nevertheless, the main goal is always the 
same: find a good trade-off between simplicity and efficiency. Indeed, 
a very abstract model simplifies the programming activity but can lead 
to a very inefficient exploitation of computing resources. Instead, a low- 
level model allows programmers to efficiently exploit the computational 
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resources but requires to programmers a tremendous effort when the 
number of resources grows. Since fhe nineties, several research groups 
have proposed fhe structured parallel programming environments (SPPE). 
Since fhe sfrucfured parallel programming model was conceived, sev¬ 
eral works have been done abouf if. Programming enviromnenfs relying 
on fhis paradigm ask programmers fo explicifly deal wifh fhe qualitative 
aspecfs of parallelism exploifafion, namely fhe application sfrucfure and 
problem decomposition sfrafegies. All fhe low-level parallelism exploifa- 
tion relafed aspecfs like communicafion, S5mchronization, mapping and 
scheduling are managed by compiler fools and run-fime supporf. In 
fhese environmenfs parallelism is exploifed by composing "skeletons", 
i.e. parallelism exploifation patterns. From language viewpoint, a skele¬ 
ton is a higher-order function that behaves as a pure function (no side- 
effects). Several real world, complex applications have been developed 
using these environments. The skeletal approach has been proved to be 
quite effective, when application algorithms can be somehow expressed 
in terms of skeleton composition. Nofwifhsfanding, skelefal program¬ 
ming has still fo make a subsfanfial impacf on mainsfream pracfice in 
parallel applications programming. 

1.1 Contribution of the thesis 

This fhesis originates from fhe wish fo address fhe issues fhaf have lim- 
ifed fhe diffusion of sfrucfured parallel programming enviromnenfs. These 
issues are well-known by fhe sfrucfured parallel programming models 
scientific communify. They have been organically reporfed in two key 
papers (UlllSl) where fhe aufhors describe bofh fhe issues and fhe fea- 
fures fhaf fhe nexf generation of sfrucfured parallel programming envi¬ 
ronmenfs have fo support in order to address them. The features "check¬ 
list" includes, as an example, the ease of use, fhe infegrafion of sfrucfured 
and unsfrucfured form of parallelizafion, the support for code reuse, 
the heterogeneity and dynamicity handling. Drawing a parallel with 
web programming model we can refer as "Skelefons 2.0" fhe nexf gen- 
erafion of sfrucfured parallel programming enviromnenfs fhaf address 
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the issues that prevent the skeleton environment to became part of the 
mainstream practice in parallel applications programming. Some groups 
of researchers involved in sfrucfured parallel programming developed 
skeleton systems fhaf have parfially addressed fhe "Skeletons 2.0" prin¬ 
ciples fo differenf degrees in difterenf combinafions. Neverfheless, fhe 
research for addressing fhe presenfed issues has jusf sfarfed. Indeed, up 
fo now, fools and models fhaf are generally recognized as fhe besf solu- 
fions for addressing fhe issues sfill do nof exisf. 

The main goal of fhis fhesis is fo presenf an organic sef of fools and 
models conceived, designed and developed fo address mosf of fhese is¬ 
sues, fherefore form fhe base of a nexf generafion skeleton system. The 
scienfific confribufion of fhe fhesis is organized in fhree main parfs. They 
reporfs four resulfs we obfained in fhe lasf fhree years. These research 
resulfs as has been already presenfed in published papers. Some resulfs 
have been achieved wifh acfual experimenfs conducfed using soffware 
fools and packages designed and developed fo fhe purpose. Some of 
them are simple, proof-of-concept tools, like JJPF dS^ or PAL (|6T). Some 
others are custom version of exisfing framework, like muskel with the 
support for developing unsfrucfured form of parallelism (|2lJ or muskel 
wifh an aspecf orienfed programming supporf (l60ll . Ofhers are parf of 
complex infernafional research projecf focused on Grid compufing, like 
the Behavioural Skeletons m- 

Our first contribution copes with the lack of models supporfing fhe 
infegrafion of unsfrucfured form of parallelizafion in skeleton systems. 
In facf, if on fhe one hand sfrucfured parallel programming environ- 
menfs raise fhe level of absfracfion perceived by programmers and guar- 
anfee good performance, on fhe ofher hand fhey resfricf fhe freedom of 
programmers fo implement arbitrary parallelism exploitation patterns. 
In order to address this issue we propose a macro data-flow based ap¬ 
proach that can be used to implement mixed parallel programming envi¬ 
ronments providing the programmer with both structured and unstruc¬ 
tured ways of expressing parallelism. Sfrucfured parallel exploifafion 
paffems are implemenfed franslafing fhem info dafa-flow graphs exe¬ 
cuted by a disfribufed macro dafa-flow inferprefer. Unsfrucfured par- 
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allelism exploitation can be achieved by explicitly programming data¬ 
flow (sub)graphs. To validate the approach, we modified a skeleton sys- 
fem fhaf in ifs original form does nof deal wifh unsfrucfured parallelism: 
muskel. We exfended muskel, in collaboration wifh fhe research sfaff 
fhaf developed if. Our cusfomized muskel is implemented exploif- 
ing (macro) dafa-flow fechnology, rafher fhan more usual skelefon fech- 
nology relying on fhe usage of implemenfafion templates. Using dafa- 
flow, fhe exfended muskel supporfs fhe developmenf of bofh classi¬ 
cal, predefined skeletons, and programmer-defined parallelism exploifa- 
fion patters. Our exfended version provides fwo mechanisms fo fhe 
muskel programmers for unsfrucfured parallelism exploifafion. Firsf, 
we provide primifives fhaf allow fo access fhe fundamenfal feafures of 
fhe dafa-flow graph generafed ouf of fhe compilation of a skelefon pro¬ 
gram. Namely, mefhods fo deliver dafa fo and refrieve dafa from dafa- 
flow graph. We provide fo programmers fhe abilify fo insfanfiafe a new 
graph in fhe fask pool by providing fhe inpuf fask foken and fo redirecf 
fhe oufpuf foken of fhe graph fo an arbifrary dafa-flow insfrucfion in fhe 
pool. Second, we provide fhe programmer wifh direcf access fo fhe def- 
inifion of dafa-flow graphs, in such a way he can describe his particular 
parallelism exploifafion pafferns fhaf cannof be efficienfly implemenfed 
wifh fhe available skeletons. The fwo mechanisms can be joinfly used 
fo program all fhose parfs of fhe applicafion fhaf cannof be easily and 
efficienfly implementing using fhe fradifional skelefons subsysfem. Un- 
forfunafely, fhis approach is nof free from shorfcomings. In facf, exploif- 
ing unsfrucfured parallelism interacting direcfly wifh dafa-flow graph 
requires fo programmers fo reason in terms of program-blocks instead of 
a monolifhic program. 

In order fo ease fhe generation of macro dafa-flow blocks and in gen¬ 
eral fo provide mechanism easing fhe use of sfrucfured parallel program¬ 
ming environmenf, we exploifed some metaprogramming fechniques. Ex¬ 
ploiting fhese fechniques fhe programmers are no longer requesfed fo 
deal wifh complex applicafion sfrucfuring buf simply fo give hinfs fo fhe 
mefaprogramming supporf using high-level directives. The direcfives 
drive fhe aufomafic applicafion fransformafion. In fhis fhesis we presenf 
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two results we obtained regarding the exploitation of metaprogramming 
techniques for parallel programming. The firsf resulf is "Parallel Absfrac- 
fion Layer" (PAL). A java annofafion based mefaprogramming frame¬ 
work fhaf resfrucfures applicafions af byfecode-level af run-fime in order 
fo make fhem parallel. The parallelizafion is obfained as5mchronously 
executing fhe annofafed mefhods. Each mefhod call is fransformed in 
a macro dafa-flow block fhaf can be dispafched and execufed on fhe 
available computing resources. PAL fransformafions depend on bofh on 
fhe resources available af run-fime and fhe hinfs provided by program¬ 
mers. The ofher resulf concerns fhe infegrafion of fhe Aspecf Orienfed 
Programming mechanisms wifh our modified muskel skeleton frame¬ 
work. We make fhis infegrafion in fwo disfincf phases, in fhe firsf phase 
we infegrafed fhe AOP mechanisms in order fo achieve very simple code 
franstormafion. In fhe second phase we implemented a more complex 
infegrafion fo obfain a supporf enabling fhe developmenf of workflows 
which sfrucfure and processing are opfimized af run-fime depending on 
fhe available compufafional resources. 

In fhis fhesis we presenf also a model fo address fwo ofher issues: fhe 
lack of supporf for code reuse, and fhe lack of supporf for handling of 
dynamicify. The muskel framework, addresses fhis lasf poinf fhrough 
fhe definition of fhe Application Manager, namely an enfify able fo ob¬ 
serve, af run-fime, fhe behavior of fhe parallel application and in case 
of faulfs or application non-funcfional requiremenf violafions if reacfs 
aiming fo fix fhe problem. The d5mamicify handling is a very imporfanf 
feafure for nexf generation parallel programming systems, especially for 
fhe ones designed for compufafional Grids. Indeed, Grid are often com¬ 
posed by heferogeneous compufer and managed by differenf adminisfra- 
fion policies. To address fhese addifional difficulfies mosf of fhe models 
and fools conceived and developed for parallel programming have fo be 
re-fhoughf and adapted. Acfually fhe muskel framework, af leasf in 
ifs original form, is designed fo be exploifed in clusfer and nefwork of 
worksfafions rafher fhan in Grids. Indeed, some of fhe implemenfafion 
choices done when if was developed limif ifs exploifafion on Grids, in 
parficular fhe ones relafed wifh communication protocol and wifh fhe 
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mechanisms for recruiting computational resource. On the other hand, 
several studies recognized that component technology could be lever¬ 
aged to ease the development of Grid Application (l25t l72ll . Indeed, a 
few component based model have been proposed by parallel comput¬ 
ing scientific community for programming Grids, as GCA (|5), CCM (l67t 
and GCM (l52t . The GCM represents one of the main European scien¬ 
tific community efforts for designing and developing (|3) a grid compo¬ 
nent model. We contributed to the design of GCM and its reference im¬ 
plementation together with the research group that developed muskel 
and with several European research groups. In particular, we focused 
our contribution on the GCM autonomic features. We referred to the 
muskel Application Manager approach, generalizing it and extending the 
approach to make it suitable for components based models. Indeed, each 
GCM component with a complete support of autonomic features has an 
Autonomic Manager that observes the component behavior. In case the 
behavior turns out to be different from the expected one the manager 
trigger a component reconfiguration. In other words, GCM autonomic 
features provide programmers with a configurable and straightforward 
way to implement autonomic grid applications. Hence, they ease the 
development of application for the Grids. Nevertheless, they rely fully 
on the application programmer's expertise for the setup of the manage¬ 
ment code, which can be quite difficult to write since it may involve the 
management of black-box components, and, notably, is tailored for the 
particular component or assembly of them. As a result, the introduction 
of d 5 mamic adaptivity and self-management might enable the manage¬ 
ment of grid d 5 mamism, and uncertainty aspects but, at the same time, 
decreases the component reuse potential since it further specializes com¬ 
ponents with application specific management code. In order to address 
this problem, we propose the Behavioural Skeletons as a novel way to 
describe autonomic components in the GGM framework. Behavioural 
Skeletons aim to describe recurring patterns of component assemblies 
that can be equipped with correct and effective management strategies 
with respect to a given management goal. Behavioural Skeletons help 
the application designer to i) design component assemblies that can be 
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effectively reused, and ii) cope with management complexity. The Be¬ 
havioural Skeletons model is an effective solution for handling dynam- 
icity, supporting reuse both of functional and non-functional code. We 
want to point out that we have not the "sole rights" concerning the Be¬ 
havioural Skeletons model. Indeed, it has been developed in conjunction 
with the other authors of the two papers about Behavioural Skeletons we 
published 


This thesis is not our first attempt of design programming model for 
parallel programming. In a previous work we developed JJPF, a Java 
and Jini based Parallel Framework, and investigated the possibilities of¬ 
fered by structured parallel programming. In (l59J we described the ar¬ 
chitecture of JJPF. JJPF was specifically designed to efficiently exploit af¬ 
fordable parallel architectures, such as a network of workstations. Its 
reactive fault-tolerance support and its d 5 mamic support for task distri¬ 
bution as well as for resources recruiting were designed to enable an 
efficient exploitation of resources in highly d 3 mamic environment. In 
particular, JJPF exploits the Jini technologies to d 5 mamically find and re¬ 
cruit the available computational resources. JJPF provide to program¬ 
mers an API enabling the development of task-parallel application fol¬ 
lowing the master-slave paradigm. It also provides an high-level sup¬ 
port for data sharing among slaves. JJPF ease the parallel programming 
task hiding most of low-level error prone issues to programmers. As we 
stated above, JJPF is implemented in Java. It simplifies the code portabil¬ 
ity among heterogeneous architectures. For the communications among 
master and slaves JJPF exploits the JERI. It is a variant of RMI allowing 
the protocol customization and as a consequence an optimization of its 
performance in several situations. For the performance purpose JJPF also 
provides an alternative to the java distributed class-loader that reduces 
the class-loading latency in some situations. Some problems encountered 
during the design of JJPF still remain open. Moreover, during the real¬ 
ization of JJPF we faced directly with the development complexity of this 
kind of software so we think that some kind of software engineering is 
needed to facilitate reuse and maintenance of source code. 
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1.2 Thesis Outline 


As we already stated, in this thesis we report our contribution to address 
the issues that are t 5 rpical of traditional structured parallel programming 
environments. The contribution is organized in three main parts. Each 
part is presented in a dedicated chapter. Moreover, there are three more 
chapters: an Introduction chapter (this one, actually), a Conclusion chap¬ 
ter and another one that introduces the problems we face in fhis fhesis 
and ouflines fhe sfafe-of-fhe-arf of exisfing solufions. In fhe remain of 
fhis secfion we describe fhe confenf of each chapfer. 


Chapter In this chapter we take into account the problems related 
to programming parallel applications, the existing solutions and their 
main limitations. In particular, after a general introduction to the differ¬ 
ent parallel programming models, the topic is focused on fhe limifafions 
fhaf prevenf fhe sfrucfured parallel programming models from spread¬ 
ing and fo become parf of fhe mainsfream pracfice. Secfion 2.1 gives a 
bird's-eye view bofh on fhe parallel archifecfures and on fhe fields in 
which parallelism has fradifionally been employed. Secfion reporfs 
a selecfion of fhe main parallel programming models disfinguishing be- 
fween fhe implicif (Secfion 2.2. 1| and explicif (Secfion 2.2.2| approaches. 
The explicif approaches are furfher discussed subdividing fhem, wifh 
respecf fo fhe absfracfion presenfed fo programmers, in high-level (Sec¬ 
fion 2.2. 3| and low-level (Secfion |2.2.4| ones. For each of fhem are pre¬ 
sented the Pros and Cons. The chapter reports also some other notable 
approaches (Section |2.2.5[ |. Then the Chapter present the structured ap¬ 
proach, an approach conceived in order to overcome the limitations of 
fraditional approaches (Secfion |2.2.6| . Some fools based on fhe sfrucfured 
parallel programming models are presenfed (Secfion |2.2.6| and ofhers are 
reported as well as references fo fhe liferafure. The models are presenfed 
highlighfing fheir feafures and main limifafions. Secfion [23] reporfs fhe 
issues fhaf nexf generafion skeleton system should own fo address fhe 
exisfing limifafions. Finally, fhe chapfer infroduces (Secfion |2.4| our con- 
fribufions fo fhe field placing fhem in fhe proper confexf, showing how 
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such contributions can be exploited for addressing the issues related to 
structured parallel programming environments. 


Chapter In this Chapter we discuss a methodology that can be ex¬ 
ploited in order to provide to programmers the possibility to mix struc¬ 
tured and unstructured ways of expressing parallelism while preserving 
mosf of fhe benefifs fypical of sfrucfured parallel programming models. 
The mefhodology is based on fhe dafa-flow model. Unsfrucfured par¬ 
allelism exploifafion is achieved by explicifly programming dafa-flow 
graphs. Secfion [TT] briefly recalls fhe sfrucfured programming models 
ouflining fheir main advanfages and limifafions. In particular, fhe sec¬ 
fion focuses on fhe skeleton customization issue. Namely fhe lack of 
flexibilify of skelefal sysfems in expressing parallel form differenf from 
fhe ones fhaf are "bundled" wifh fhe skeleton framework and fheir com- 
posifions. Then fhe secfion infroduces fhe macro dafa-flow based ap¬ 
proach we conceived in order to address of fhis limifafion and reporfs 
fhe relafed work: alfernafive approaches addressing fhe sfrucfured par¬ 


allel programming limifafions. Secfion 3.2 infroduces bofh fhe classical 
femplafe-based implemenfafion of skelefon sysfems and fhe more recenf 


dafa-flow technologies based one used in muskel. Secfion 3.3.1 de¬ 
scribes fhe defails of our confribufion, i.e. how we exploited fhe mefhod¬ 
ology presented to exfend fhe muskel framework. Finally, Secfion 3.4 
reporfs fhe experimenfal resulfs we obfained conducfing some fesf using 
our customized muskel framework. 


Chapter|^ In fhis Chapfer we infroduce some novel mefaprogramming 
fechniques for fhe generafion and opfimizafion of macro dafa-flow blocks 
This Chapfer presenfs our efforfs aimed af providing mefaprogramming 
fools and models for optimizing af run-fime fhe execufion of sfrucfured 
parallel applications. The approaches are based on fhe run-fime gener¬ 
afion of macro dafa-flow blocks from fhe application code. The Chap¬ 
fer discusses how we exploifed fhese fechniques bofh in our modified 
muskel framework as well as in ofher frameworks we developed. Sec¬ 
fion |4^ presenfs fhe mofivafions behind our confribufions. Secfion 4.2 
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presents PAL, our first result in the field. The core of PAL framework 
is ifs mefaprogramming engine fhaf fransforms af run-fime an armo- 
fafed sequenfial java code in a parallel program exploding bofh program¬ 


mer hinfs and information abouf executing plafforms. Section 4.2.1 de- 


4.2.2 


scribes fhe defails of our PAL profofype implemenfafion. Secfion 
reporfs fhe experimenfal resulfs we obfained fesfing PAL framework. 


Secfion 4.2.3 discusses fhe mofivafions fhaf convinced us fo infegrafe 


fhe PAL approach fo our modified muskel framework. Secfion 14.3 
describes fhe preliminary affempfs we made infegrafing mefaprogram¬ 
ming fechniques in muskel showing how Aspecf Orienfed Program¬ 
ming can be exploifed fo do some simple code fransformations. Secfion 


4.4 describes how we furfher enhanced muskel making if able fo exploif 


mefaprogramming for run-fime code optimizations. In particular, how if 
can be exploifed fo opfimize fhe parallel execution of compufafions ex¬ 


pressed as workflows. Secfion 4.4.2 describes fhe implemenfafion defails 
of workflows fransformafions and Secfion |4.4.3| presenfs fhe resulfs of 
some experimenfs we conducfed. Finally Secfion [43] presenfs a compar¬ 
ison of fhe two approaches. 


Chapter]^ In this Chapter we present some results about the customiza¬ 
tion of skeletons applied fo fhe Grid Componenf Model. In fhis chapfer 
we presenf fhe Behavioural Skelefons model, an approach, we confribufe 
fo conceive and validafe, aimed af provide programmers wifh fhe abilify 
fo implemenf autonomic grid componenf-based applications complefely 
faking care of fhe parallelism exploifafion defails by simply insfanfiafing 
existing skelefons and by providing suifable, functional paramefers. The 
model has been specifically conceived fo enable code reuse and dynam- 
icify handling. Secfion [ST] describes how componenf-based applications 
can ease fhe fask of developing grid applicafions. Secfion [5^ ouflines fhe 
grid componenf model focusing on ifs aufonomic feafures. After, Secfion 
|5.4| presenfs fhe Behavioural Skelefons model, Secfion [53] reporfs a sef of 
nofeworfhy Behavioural Skelefons and Secfion |5.6| describe fheir GCM 


implemenfafion. Secfion 5.7 describes a sef of experimenf we conducfed 
fo validafe fhe Behavioural Skelefons model. 
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Chapter|^ This Chapter summarizes the materials contained in the pre¬ 
vious chapters and discusses the conclusions of the thesis. Finally, the 
future work related to the thesis is introduced. 
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Chapter 2 

High-Level Parallel 
Programming 


As we already stated in the Introduction, using several processors (or 
computational resources) at the same time (in parallel), however, intro¬ 
duces some difficulties. The conceptual barrier encountered by the pro¬ 
grammers in efficiently coordinating many concurrent activities towards 
a single goal is an example of such barriers. To address these difficulties 
software developers need high-level programming models for sensibly 
raising the abstraction of computational resources. This is a fundamen¬ 
tal requirement to avoid programmers having to deal with low-level co¬ 
ordination mechanisms. In fact, low-level parallel programming is an 
error prone approach that distracts programmers from qualitative as¬ 
pects of parallelization. Throughout the ages, researchers conceived and 
developed several models for high-level parallel programming. How¬ 
ever, most of current implementations of very high-level programming 
models often suffer from low performance. This because of the abstrac¬ 
tion penalty, which actually has historically limited the usage of high- 
level programming techniques in high performance computing. For this 
reason, nowadays most of parallel programs are developed exploiting 
lower-level language, even if a higher-level language would make the 
coding easier. Structured parallel programming models were conceived to 
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be an alternative both to very high-level models and to low-level mod¬ 
els. Structured parallel programming models ask programmers to explic¬ 
itly deal with the qualitative aspects of parallelism exploitation, namely 
the application structure and problem decomposition strategies. Com¬ 
pilers and run-time supports manage all the low-level parallelism ex¬ 
ploitation related aspects like communication, S5mchronization, schedul¬ 
ing and mapping. The Structured Way is driven by those two observa¬ 
tions: there are some things programmers do better than compilers, and 
there are some things that compilers do better than programmers. Nev¬ 
ertheless, also the structured models are not perfect and free from limi- 
fafions. In facf, for some years researchers very expert in structured par¬ 
allel programming models have outlined the features that the next gen¬ 
eration of sfrucfured models have fo provide in order fo address fhese 
limifafions (misD. In nexf fhree chapfers of fhis fhesis we present some 
results we obtained as an attempt of address some of fhese limifations. 


Chapter road-map The chapter starts with a bird's-eye view both on the par¬ 
allel architectures and on the fields in which parallelism has traditionally been 
employed (Section^Zl^. Then, it reports the main parallel programming mod¬ 


els (Section 2.2 ' distinguishing between the implicit (Section 2.2.1 ' and explicit 
(Section 2.2.2\ approaches. The explicit approaches are further subdivided in 
high-level (Section 2.2.3 f and low-level (Section 2.2.41 ones. The chapter re¬ 
ports also some other notable approaches (Section 2.2.51 Then the Chapter 
present the structured approach, an approach conceived in order to overcome 
the limitations of traditional approaches (Section 2.2.6 1 Some tools based on the 
structured parallel programming models are presented (Section ^.2.6 ' highlight¬ 
ing their features and main limitations. Then Section ^f^ reports the issues that 
next generation skeleton system should own to address the existing limitations. 
Finally, the chapter introduces (Section 2.4' our contributions to the field plac¬ 
ing them in the proper context, showing how they can be exploited for addressing 
some of the issues related to structured parallel programming environments. 
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2.1 From sequential to parallel architectures 


The Von Neumann architecture is a very common and well-known com¬ 
puter design model. It has a very simple formulation and can be de¬ 
scribed as a sequential process running in a linear address space. It con¬ 
sists in a processing unit and a single separate storage structure to hold 
both instructions and data. The Von Neumann model "implements" a 
universal Turing machine. It represents the common "referential model" 
of specifying sequential architectures, in confrasf wifh parallel architectures. 
In a parallel archifecfure many insfrucfions are carried ouf simulfane- 
ously. Parallel compufers operafe on fhe principle fhaf large problems 
can almosf always be divided info smaller ones, which may be carried 
ouf af fhe same fime. Parallel archifecfures exisf in several forms and lev¬ 
els. They range from superscalar processors fo compufafional Grids.In 
fhis section we briefly mention some of fhe mosf common forms of par¬ 
allelism, wifhouf claiming fo be exhaustive buf only fo give an idea of 
fhe variefy of fhe exisfing forms of parallelism. 


Bit-level parallelism is a form of parallelizafion based on increasing 
processor word size. If leads fo a reduction of fhe number of insfrucfions 
fhe processor musf execufe in order fo perform an operafion on vari¬ 
ables whose sizes are greafer fhan fhe lengfh of fhe word. (For insfance, 
consider a case where a 16-bif processor musf add fwo 32-bif numbers. 
The processor musf firsf add fhe 16 lower-order bifs from each number, 
and fhen add fhe 16 higher-order bifs, and fhe carry from fhe previous 
add requiring fwo insfrucfions fo complefe a single operafion. A 32-bif 
processor would be able fo complefe fhe operafion using a single insfruc- 
fion). Historically 4-bif microprocessors were replaced wifh 8-bif, fhen 
16-bif, fhen 32-bif microprocessors. This frend generally came fo an end 
wifh fhe infroducfion of 32-bif processors, which has been a sfandard 
in general purpose computing for fwo decades. Only recenfly wifh fhe 
proliferafion of processors based bofh on fhe IBM PowerPC G5 proces¬ 
sor and on fhe x86-64 archifecfures, fhe 64-bif processors have become 
commonplace. 
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Instruction-level parallelism is a form of parallelization based on fhe 
simulfaneous execution of insfrucfions parf of a compufer program. Even 
if ordinary programs are f 5 rpically wriffen according fo a sequential ex¬ 
ecution model where insfrucfions execufe one affer fhe ofher and in fhe 
order specified by fhe programmer, in some significanf cases fhere is no 
need fo follow fhis order. ILP allows fhe compiler and fhe processor fo 
overlap fhe execufion of mulfiple insfrucfions or even fo change fhe or¬ 
der in which insfrucfions are execufed. Due fo ifs nafure, ILP requires 
an hardware supporf; micro-archifecfural fechniques fhaf are used fo ex- 
ploif ILP include (for a beffer descripfion see (I104H : 

• Insfrucfion pipelining, where fhe execufion of mulfiple insfrucfions 
can be partially overlapped. 

• Superscalar execufion, in which mulfiple execufion unifs are used 
fo execufe mulfiple insfrucfions in parallel. In f 5 q)ical superscalar 
processors, fhe insfrucfions execufing simulfaneously are adjacenf 
in fhe original program order. 

• Ouf-of-order execufion, where insfrucfions execufe in any order 
fhaf does nof violafe dafa dependencies. Nofe fhaf fhis fechnique 
is orfhogonal w.r.f. bofh pipelining and superscalar. 

• Regisfer renaming, which refers fo a fechnique used fo avoid un¬ 
necessary serializafion of program operations imposed by fhe reuse 
of regisfers by fhose operations, used fo enable ouf-of-order execu¬ 
fion. 

• Speculative execufion, which allows fhe execufion of complefe in¬ 
structions or parts of insfrucfions before being cerfain whefher fhis 
execufion should fake place or nof. A commonly used form of spec¬ 
ulative execufion is confrol flow speculafion where insfrucfions fol¬ 
lowing a confrol flow insfrucfion (e.g., a branch) are execufed be¬ 
fore fhe fargef of fhe confrol flow insfrucfion is defermined. Several 
ofher forms of speculafive execufion have been proposed and are 
in use including speculafive execufion driven by value prediction, 
memory dependence predicfion and cache lafency predicfion. 
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• Branch prediction, which is used to avoid stalling for control de¬ 
pendencies to be resolved. Branch prediction is used with specula¬ 
tive execution. 

Data parallelism is a form of parallelizafion of compufer code across 
multiple processors in parallel computing environmenfs. This paradigm 
is useful for faking advanfage of fhe large amounfs of dafa parallelism 
fhaf is available in many scienfific/numeric applicafions. The dafa par¬ 
allelism is exploifed by performing fhe same operation on a large amounf 
of dafa, disfribufed across fhe processors of fhe machine. From fhe pro¬ 
grammer viewpoinf, languages based on dafa-parallel paradigm (such as 
HPF, skefched in Secfion [2.2.5| are pretty similar fo sequential languages. 
The main difference is fhaf cerfain dafa f 5 rpes are defined fo be parallel. 
Parallel dafa values consisf of a collecfion of sfandard, scalar dafa values. 

The dafa-parallel paradigm has some main virfues fhaf have led fo ifs 
success. Parallel dafa f 5 rpes are fypically sfafic in size (e.g. arrays); fheir 
disfribufion across fhe machine is usually done af compile-fime. Any 
S5mchronizafion or communication fhaf is needed fo perform an opera- 
fion on a parallel value is aufomafically added by fhe compiler /run-fime 
sysfem. The processors collectively compufe operations on parallel dafa 
values; compulation load usually disfribufed direcfly linking dafa val¬ 
ues and compulations fhrough fhe owner compufes rule. As dafa values, 
compulation load is sfafically disfribufed across fhe processors of fhe sys¬ 
fem. The dafa parallelism approach f 5 rpically offers very good scalabilify. 
Because operations may be applied idenfically fo many dafa ifems in par¬ 
allel, fhe amounf of parallelism is dicfafed by fhe problem size. Higher 
amounfs of parallelism may be exploifed by simply solving larger prob¬ 
lems wifh greafer amounfs of compulation. Dafa parallelism is also sim¬ 
ple and easy fo exploif. Because dafa parallelism is highly uniform, if 
can usually be aufomafically defecfed by an advanced compiler, wifhouf 
forcing fhe programmer fo manage explicifly processes, communication, 
or synchronizafion. Many scienfific applicafions may be nafurally spec¬ 
ified in a dafa-parallel manner. In fhese settings, programs dafa layouf 
is often fixed; fhe mosf used dafa sfrucfures are large arrays. Operations 
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on whole data structures, such as adding two arrays or taking the inner 
product of two vectors, are common, as are grid-based methods for solv¬ 
ing partial differential equafions (PDFs). In spife of fhis, dafa parallelism 
has a significanf drawback: fhe limifed range of applicafions for which 
dafa-parallel is well suifed. Applicafions wifh dafa parallelism fend fo be 
sfafic in nafure; fhe confrol flow of a dafa-parallel program is mosfly dafa 
independenf. Many applicafions are more d 5 mamic in nafure and do nof 
have fhese characferisfics. To run in parallel, fhese d 5 mamic applicafions 
need fo perform independenf operafions af fhe same time. These ap¬ 
plicafions, which may be as simple as recursively computing Fibonacci 
numbers or as complex as compufer chess and n-body simulations, are 
nearly impossible parallelize using dafa parallelism. 

Task parallelism is a form of parallelizafion of compufer code across 
multiple processors in parallel computing environmenfs. Task paral¬ 
lelism focuses on disfribufing execufion processes across differenf paral¬ 
lel computing nodes. In fhe fask-parallel paradigm fhe program consisfs 
of a sef of (pofenfially disfincf) parallel fasks fhaf inferacf fhrough explicif 
communication and S 5 mchronizafion. Task parallelism may be bofh S 5 m- 
chronous and as 5 mchronous. A major advanfage of fask parallelism is ifs 
flexibilify. Many scienfific applicafions confain fask parallelism. For ex¬ 
ample, in a climafe model application fhe afmospheric and ocean circu¬ 
lation may be compufed in parallel. A fask-parallel language can express 
fhis relationship easily, even if differenf mefhods are used for fhe fwo cir- 
culafion models. Anofher nafural application of fask-parallel languages 
is reactive sysfems in which fasks musf produce oufpuf in response fo 
changing inpufs, in a fime-dependenf manner. Anofher common sfruc- 
fured paradigm exploifs parallelism on differenf dafa ifems fhrough fask 
replication. For example, fhe elaboration of a video sfream may involve 
fhe filfering on each single frame. In a fask-parallel language fhe fiber 
may be farmed ouf by spreading differenf frames on differenf worker 
processes, each of fhem compufing fhe same funcfion. In fhe fask par¬ 
allelism approach fhe inferacfions befween fasks are explicif, fhus fhe 
programmer can wrife programs fhaf exploif parallelism nof defecfable 
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automatically by compiler techniques. In general, task parallelism is less 
dependent on advanced compiler technology than the data parallelism; 
in many cases, all that is strictly necessary is the translation of task in¬ 
teractions into appropriate low-level primitives on the target architec¬ 
ture. A disadvantage of fhe fask-parallel programming model is fhaf 
if requires exfra efforf from fhe programmer fo creafe explicif parallel 
fasks and manage fheir communicafion and S 5 mchronizafion. Because 
communicafion and S5mchronizafion are explicif, changing fhe manner a 
program is parallelized may require exfensive modificafions fo fhe pro¬ 
gram fexf. 

Due fo fheir nafure data and task parallelism (unlike the bit level and 
instruction level parallelism) cannot be fruitfully exploded using a sin¬ 
gle CPU sysfem buf fhey are well-failored for mulfi-processors or clusfer 
compufers, f 5 rpically referred as parallel compufers. 

For many years parallel compufers has been mainly used in high 
performance compufing, buf fhey have spread in recenf years as con- 
venienf and effecfive way fo increase fhe compufafional power of per¬ 
sonal compufers and worksfafions due fo physical consfrainfs prevenf- 
ing frequency scaling of CPUs. Hence, parallel archifecfures are becom¬ 
ing fhe dominanf paradigm in compufer archifecfure, mainly in fhe form 
of mulficore processors (|28]| . Indeed, if a problem requires a huge com¬ 
pufafional capacify fo be rapidly solved and such a power cannof be ob- 
fained using a single processing element (PE) fhe only suifable solufion is 
fo use many processors simulfaneously. Traditionally, parallel architec¬ 
tures have been motivated by numerical simulations of complex sysfems 
and "Grand Challenge Problems" such as: 

• weafher and climate forecasting 

• chemical and nuclear reactions simulations 

• biological, human genome analysis 

• geological, seismic activity analysis 

• mechanical devices and electronic circuits' behavior simulations 
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Today, also commercial applications need the development of faster and 
faster computers. These applications require to process large amounts of 
dafa in sophisficafed ways. Example applicafions include: 

• parallel dafabases, dafa mining 

• web search engines, web based business services 

• compufer-aided medical diagnosis 

• managemenf of national and multi-national corporations 

• advanced graphics and virfual realify particularly in fhe enferfain- 
menf indusfry 

• networked video and mulfi-media fechnologies 

• collaborative working environmenfs 

Unforfunafely, as we already sfafed before, using several PEs af fhe same 
fime infroduces some difficulties. Among fhe ofhers: (i) code and dafa 
have fo be decomposed and disfribufed among fhe compufafional re¬ 
sources, (ii) work and communicafions of resources have fo be simulfa- 
neously coordinafed and (iii) faulf-folerance has fo be managed. Thus, 
fhe design and implemenfafion of software sysfems fhaf can ease fhis 
burden is very imporfanf. Indeed, since fhe early sfeps of compufer 
science, researchers conceived and designed programming models, sys¬ 
fems and fools aiming af supporting fhe developmenf of parallel appli¬ 
cafions. Such sysfems musf find a good balance between the simplic¬ 
ity of fhe inferface presenfed fo fhe programmers and fheir implemen¬ 
fafion efficiency, binding a good frade-off is a grand challenge. Indeed, 
a very absfracf model simplifies fhe programming acfivify buf can lead 
fo a very inefficienf exploifafion of compufing resources. Insfead, a low- 
level model allows programmers fo use efficienfly fhe compufafional re¬ 
sources buf requires fremendous efforfs from fhe programmers when fhe 
number of resources grows. 


20 



2.2 Parallel programming models: 
State-of-the-art 

A good way to organize the state of art of parallel programming mod¬ 
els for reporting purpose is to divide them with respect to their level of 
absfracfion. Therefore, in fhis section we reporf a selecfion of fhe main 
parallel programming models, proposed by compufer scienfisf over fhe 
years, classifying fhem wifh respecf fo fhe level of absfracfion provided 
fo programmers. Wifh respecf fo fhis aspecf, fhe parallel programming 
models can be roughly partitioned in two main classes: fhe implicif par¬ 
allel models and fhe explicif ones. The former complefely cover up par¬ 
allelism fo programmers. Typically, fhey are exploded by functional and 
logic languages. The latter ask programmers fo deal direcfly wifh par¬ 
allelism. These models can be furfher partitioned, w.r.f. fhe absfracfion 
perspecfive, in fhree cafegories: high, medium and low-level program¬ 
ming models. 

In the remaining of fhis secfion we describe for each cafegory by way 
of examples, some programming models and fools belonging fo if show¬ 
ing fhe models Pros & Cons. In particular, Secfion |2.2.1 describes fhe 
functional and logic models as an example of implicif models for paral¬ 


lel programming, Secfion 2.2.3 shows fhe data-flow model as a represen- 


fafive of high-level explicif models. In Secfion 2.2.4 we oufline fhe low- 
level approaches describing fhe OpenMP and MPI frameworks. Then, 


in Secfion 2.2.5 we reporf some ofher nofable approaches. Finally, we 


describe fhe sfrucfured approach in Secfion 2.2.6 if is one of fhe main 
medium-level models. Here we describe also some our pasf confribu- 
fions in fhe field (Secfion |2.2.6[ |. 


2.2.1 Implicit approaches 

These sysfems present to programmers a programming model entirely 
devoid of parallelism and complefely isolafed from fhe underlying im- 
plemenfafion mechanism. Such sysfems fypically presenf funcfional or 
logical models of compufafion. They are often referred fo as being "declar- 
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ative" systems, since the programmer makes a series of declarations defin¬ 
ing fhe properfies of a solufion fo some problem, rafher fhan specifying 
a precise series of operations which will lead fo fhe solufion. Thus, lan¬ 
guages of fhis t 5 rpe are neifher parallel nor sequential, having no nofion 
af all of a flow of confrol. 

Funcfional languages are based on fhe lambda calculus. If is a very 
simple, buf powerful language fo define expressions and fheir fransfor- 
mafion rules. The only objecfs presenf are identifiers, single argumenf 
function definifions ("absfracfions") and applications of funcfions fo ar- 
gumenfs. A "program" consisfs of a collection of such objecfs. The pro¬ 
gram execution is performed applying a fop-level function fo an argu¬ 
menf. This t 5 q)e of function applicafion is fhe only operation presenf 
and involves fhe replacemenf of a funcfion-argumenf pair wifh a copy 
of fhe function body (from ifs definition) in which occurrences of fhe 
"free" variable have been replaced by copies of fhe acfual argumenf. This 
simple sysfem can be shown fo provide as much compufafional power 
as any ofher fundamenfal computing mechanism (e.g. fhe Turing ma¬ 
chine). A particularly powerful aspecf of fhe model is fhe abilify fo define 
"higher order funcfions", namely, funcfions faking funcfions as inpuf pa- 
ramefer. Ofher convenienf feafures such as multiple argumenf funcfions, 
localized definifions and dafa sfrucfures may all be defined as lambda 
expressions. 

In fhe same way, a high-level funcfional program is simply a func¬ 
tion definifion fhaf refers fo ofher funcfions in ifs body. A "call" of fhe 
program involves supplying argumenfs fo fhis funcfion and "execution" 
consisfs of using fhe funcfion definifions (concepfually using fhe appli¬ 
cafion by subsfifufion fechnique from fhe lambda calculus) fo obfain an 
alfernafive, buf equivalenf represenfafion of fhe funcfion and argumenfs 
pair, namely a more useful represenfafion of fhe original program and 
fhe "inpuf". 

The key poinf of fhis approach is fhaf execufion may progress from 
fhe initial fo fhe final represenfafion in any fashion fhaf preserves fhe 
equivalence. In particular, if will offen be possible fo execufe many frans- 
formafion sfeps concurrenfly since fhe convenfional problems associafed 
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with changes of state have been discarded along with the notions of state 
and store themselves. A quite common way to represent the program as 
it evolves is as a graph, in which nodes represent function applications 
and the children of a node are the ("input") arguments of the correspond¬ 
ing application. The process of expanding and contracting the graph is 
referred to as "graph reduction". 

Exploiting this approach, the parallelization via decomposition is sim¬ 
ple. The abstract execution model allows candidate nodes to be expanded 
at any time, while function applications may be evaluated as soon as ar¬ 
guments are available.Thus, a potentially parallel process is generated 
every time a node reaches one of these states. 

It is important to realize that this approach does not imply that every 
functional program is a highly parallel one. As a trivial, well-known, 
example, consider defining a function to compute factorials. 

The obvious definition will look something like this: 

factorial 0 = 1 

factorial n = n X factorial (n — 1) 


Such a function would execute in a sequential way on a t 5 q)ical graph 
reduction machine, irrespective of the number of available processors. A 
more complex definition notes that 


factorial 0 
factorial n 
product a a 

product a b 


1 

product 1 n 
a 

(product a [ — ^ — J) X (product ([ — ^ — J -|- 1) b) 


This definition produces significant potential parallelism. Although 
declarative systems involve no explicit notion of execution sequence, it 
is unfortunately clear that, in order to optimize the parallel execution 
programmers must be aware of the execution mechanisms. 
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An alternative approach recognizes the difficulty of aufomafing dis- 
fribufion process and infroduces program annofafions fhaf programmers 
exploif fo drive fhe execution mechanism in order fo improve ifs effi¬ 
ciency. Such additions may be argued fo move fhe model ouf of fhis caf- 
egory, in fhaf fhe programmer is now parfially responsible (and aware) 
for fhe fask of parallel decomposition. Similarly, dSTIl discusses a lan¬ 
guage which allows program partitioning and inferconnecfion sfrucfure 
fo be described in a declarative sfyle. 

Anofher cafegory of implicif sysfems consisfs in parallel logic lan¬ 
guages. They are based on Horn clauses, a resfricfed form of firsf or¬ 
der logic. The compufafional model focuses on fhe definition and in- 
vesfigafion of relafionships described as predicafes, among dafa objecfs 
described as inpuf argumenfs fo fhese predicafes. As in funcfional pro¬ 
gramming, fhe specificafion of a compufafion consisfs of a collecfion of 
predicafes and clauses. In fhe logic model fhe role of fhe oufermosf func- 
fion application, is played by fhe oufermosf predicafe fogefher wifh ifs 
argumenfs. The argumenfs inferprefafion is similar: "execution" con¬ 
sisfs of deciding whefher fhe predicafe is frue given fhe argumenfs and 
fhe associafed definifions. Furfhermore, if is possible fo specify fhe ouf¬ 
ermosf predicafe wifh unbound argumenfs fo find bindings fo fhe argu¬ 
menfs fhaf allow fhe predicafe fo be safisfied, or fo defermine fhaf no 
such bindings exisf. 

Af an absfracf level, fhe process of evaluation may be seen as expand¬ 
ing and searching a free of possibilifies presenfed by consideration of fhe 
various dependencies befween appropriafe predicafes and clauses. As 
wifh graph reducfion, fhe semantics of pure logic languages offen allow 
fhis process fo proceed af many poinfs in parallel. Four principal kinds 
of (implicifly exploifable) parallelism can be idenfified in logic programs: 

• Unification parallelism arises when argumenfs of a goal are unified 
wifh fhose of a clause head wifh fhe same name and arify. The 
differenf argumenf ferms can be unified in parallel as can fhe dif- 
ferenf subferms in a ferm (l34t . Unification parallelism is very fine¬ 
grained and has been exploifed by building specialized processors 
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with multiple unification units. 

• Or-parallelism arises when more than one rule defines some relafion 
and a procedure call unifies wifh more fhan one rule head; fhe cor¬ 
responding bodies can fhen be execufed in parallel wifh each ofher. 
Or-parallelism is a way of efficienfly searching for solufions fo fhe 
query, by exploring alfernafive solufions in parallel. 

• Independent and-parallelism arises when more fhan one goal is presenf 
in fhe query or in fhe body of a procedure, and fhe run-fime bind¬ 
ings for fhe variables in fhese goals are such fhaf fwo or more goals 
are independenf of one anofher, i.e., fheir resulfing argumenf ferms 
affer applying fhe bindings of fhe variables are eifher variable-free 
or have non-infersecfing sefs of variables. Parallel execufion of 
such goals resulf in and-parallelism. 

• Dependent and-parallelism arises when fwo or more goals in fhe body 
of a procedure have a common variable and are execufed in par¬ 
allel. Dependenf and-parallelism can be exploifed in fwo ways: 
(i) fhe fwo goals can be execufed independenfly unfil one of fhem 
accesses/binds fhe common variable, (ii) Once fhe common vari¬ 
able is accessed by one of fhe goals, if if is bound fo a strucfure, 
or sfream (fhe goal generafing fhis binding is called fhe producer), 
and fhis sfrucfure is read as an inpuf argumenf of fhe ofher goal 
(called fhe consumer) fhen parallelism can be furfher exploifed by 
having fhe consumer goal compufe wifh one elemenf of fhe sfream 
while fhe producer goal is compufing fhe nexf elemenf. Case (i) is 
very similar fo independenf and-parallelism. Case (ii) is somefimes 
also referred fo as sfream-parallelism and is useful for speeding up 
producer-consumer infer actions. 

Figurej^show a simple program for compufing fhe Fibonacci number. 
The fwo lisfs of goals, each enclosed wifhin square brackefs above, have 
no dafa-dependencies among fhemselves and hence can be execufed in¬ 
dependenfly in parallel wifh each ofher. However, fhe lasf subgoal N is 
N1 + N2 depends on fhe oufcomes of fhe fwo and-parallel subgoals, and 


25 


fib(0, 1) . 
fib(1, 1) . 

fib(M, N) [ Ml is M - 1, fib(Ml, Nl) ], 

[ M2 is M - 2, fib(M2, N2) ], 
N is Nl + N2. 


Figure 1: Fibonacci program parallelizable with independent and- 
parallelism 


should start execution only after Nl and N2 get bound. Consider that, as 
in case of funcfional languages, fhe programmers in order fo exploif fhe 
pofenfial applicafion parallelism should give a proper sfrucfure fo fhe 
program. 

If should be poinfed ouf fhaf exisf some exfensions for logic program¬ 
ming language wifh explicif consfrucfs for concurrency. They can be 
largely puf info fhree cafegories: 

• fhose fhaf add explicif message passing primifives fo Prolog, e.g., 
Delfa Prolog (|95ll and CS-prolog dTTt . Multiple Prolog processes are 
run in parallel fhaf communicafe wifh each ofher via messages. 

• fhose fhaf add blackboard primifives fo Prolog, e.g.. Shared Prolog 
(l46t . These primifives are used by mulfiple Prolog processes run¬ 
ning in parallel fo communicafe wifh each ofher via fhe blackboard. 

• fhose based on guards, committed choice, and dafa-flow S 5 mchro- 
nizafion, e.g., Parlog (l48t , GHC and Concurrenf Prolog (I102II . 

As for fhe funcfional languages, fhe exfensions of parallel logic languages 
move fhe approach oufside fhe cafegory of implicif parallel program¬ 
ming models. 

Similarifies befween funcfional and logic sfyles are emphasized in 
([65). 


Summarizing Pros and Cons Implicit parallel programming models pro¬ 
vide programmers a very expressive programming metaphor: programmers can 
implement parallel application without actually deal with parallelism. Unfor¬ 
tunately, this ease is paid in terms of efficiency. In order to address such per- 
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formance issues researchers introduced some annotation mechanisms and com¬ 
munication primitives, through which programmers can drive the code paral¬ 
lelization. Nevertheless, such additions place the model out of highly abstract 
systems category because the programmer exploiting annotations is partly re¬ 
sponsible and aware for the task of decomposition. 

2 . 2.2 Explicit models 

The inefficient exploitation of available parallelism caused by the ab¬ 
sence of parallel structure in implicit parallel programs is the main rea¬ 
son why explicit parallel programming models exist. These models are 
based on the assumption that programmers are often the best judges of 
how parallelism can be exploited for a particular application. Actually, 
in nearly every case the use of explicit parallelism will obtain a better 
efficiency than implicit parallelism models. 

2.2.3 High-level explicit models: data-flow 

The models belonging to this category still not require programmers to 
deal with the several issues related with parallel programming. For in¬ 
stance communications, fault-tolerance, heterogeneity, data decomposi¬ 
tion and task granularity. Programmers are only required to write their 
applications as a set of independent instructions that interact each other 
through well-known interfaces, so that automatic tools can execute it in 
parallel. The data-flow model of computation is the main representative 
of this class of models. 

In the data-flow model (for a deep description see (l26t IHOl 110311115H 
the computations are represented by a graph of "operator" or "instruc¬ 
tion" nodes connected by edges along which data items flow. Each node 
receives by its input edges the data "tokens", it performs some simple, 
stateless, calculation and distributes resultant data tokens on its output 
edges. A node may only perform its operation once it has received all the 
data tokens required, from all of its inputs. Thus, each node may com¬ 
pute in parallel, subject only to the availability of data. The processes of 
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associating output tokens with appropriate operator nodes and of decid¬ 
ing which are ready for execufion is known as "mafching" process. 

Under fhis paradigm fhere is no currenf operafion, and each operafor 
is free fo execufe when all ifs inpuf fokens are available. The model is 
nafurally concurrenf, and fhe concurrency grain depends on fhe opera- 
fions grain. 

The dafa-flow model has fhe single-assignmenf properfy. Values are 
dafa fokens fhaf are carried from fheir producing node fo fhe node fhaf 
consumes fhem; there is no concept of a variable wifh a sfafe fhat can be 
arbifrarily updafed lafer. In dafa-flow, identifiers may be used fo name 
these data tokens. Such identifiers are fhus eifher undefined (nof yef pro¬ 
duced) or carry a single unique value; fhey cannof be updafed. A node 
wifh all inpuf dafa available is called "fireable". When a node is "fire- 
able" is ready fo be run on a dafa-flow interprefer. Each dafa-flow infer- 
prefer is called "acfor". The feafures of a dafa-flow model were lisfed by 
Ackerman in ifs 1982 milesfone paper dlOt . They are: 

• side effecfs free; 

• localify of effecf; 

• equivalence of insfrucfion scheduling wifh dafa dependencies; 

• single-assignmenf semantics; 

• an unusual nofafion for iferafions; 

• lack of hisfory sensifivify in procedures. 

S 5 mchronizafion is aufomafically provided by fhe foken transport mech¬ 
anism. Parallelism is exploited in data-flow archifecfures by allowing 
any acfor fo execufe on any processor and by allowing as many enabled 
acfors fo fire as fhere are processors fo execufe fhem. When fhere are a 
sufficienfly large number of processors, only acfors fhat do not have the 
input data available are not enabled. 

A key feature of fhe model is fhaf fhe order of acfor execufion does nof 
affecf fhe resulf. Thus, fhe dafa-flow model nafurally achieves high de¬ 
grees of parallelism. Neverfheless, fradifional dafa-flow presenfs fhree 
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major problems when considered for large distributed (grid) environ¬ 
ments. 

• The granularity of traditional data-flow is too small for many dis¬ 
tributed architectures, for instance related to distributed memory 
access time (where latencies are measured in hundreds to thou¬ 
sands of microseconds). The overhead of token transport and actor 
scheduling and instantiation requires that the granularity of com¬ 
putation be at least hundreds of thousands, and perhaps million of 
instructions. 

• The programming abstraction provided to programmers is quite 
different with respect to the traditional sequential one. 

The main difference between this approach and those discussed above 
is that whereas a graph reducer manipulates the graph by modifying 
both data and the "instruction code" itself, a data-flow graph is statically 
defined by the program and only data is manipulated. 

Data-flow based languages may be dressed up to resemble sequen¬ 
tial imperative languages (1271 , particularly in case of "scientific" appli¬ 
cations. The compilation process from high-level language to the un¬ 
derlying data-flow graph is quite similar to the process of expansion in 
graph reduction. It is equivalent to the decomposition phase of parallel 
implementation. 

All the problems of distribution, communication and S 5 mchronization 
are associated with the data-flow graph and the interactions between its 
node operators. Although the structure of the graph is static, it will only 
be apparent during (or even after) execution that some sections of the 
graph were more active than others. Thus, a good distribution scheme is 
difficult to obtain without any additional information, for instance in the 
form of programmer annotations. 

Macro-Dataflow approaches 

The macro data-flow model extends the traditional data-flow model ad¬ 
dressing its main problems. There are two principal differences with 
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traditional data-flow. First, the granularity of the actors is considerably 
larger (indeed in this case they are named "macro" actors). This allows 
to achieve a good scalability when the degree of parallelism, namely 
fhe number of recruifed PEs, increases. Second, some acfors m can 
mainfain sfafe informafion befween firings, providing an effective way 
fo model side-eflecfs and non-deferminism, fhese acfors are called "per- 
sisfenf" acfors. Some examples of exisfing and widely used macro-acfors 
implemenf high-level functions such as: mafrix mulfiplicafion, Gaussian 
elimination or image convolution instead of individual machine insfruc- 
tions. Macro acfors can be described as follows. 

Regular actors are similar to actors in the data-flow model. Specifically, 
all regular acfors of a given f 5 rpe are functionally equivalenf. A regular 
acfor is enabled and may execufe when all of ifs inpuf fokens are avail¬ 
able. If performs some compufafion, generafing outpuf fokens fhaf de¬ 
pend only on ifs inpuf fokens. If may mainfain infernal sfafe informafion 
during fhe course of a single execution, buf no sfafe information is pre¬ 
served from one execufion to another; regular actors, therefore, represenf 
pure funcfions. 

Persistent actors maintain state information that is preserved from one 
execufion fo fhe nexf. Outpuf fokens generafed by a persisfenf acfor dur¬ 
ing differenf execufions are nof necessarily fhe same for fhe same inpuf 
fokens. The sfafe corresponds fo member variables (insfance variables) 
in fhe objecf-oriented paradigm. This correspondence implies fhat sev¬ 
eral differenf acfors may share fhe same sfafe, (as an example wifh fhe 
enqueue and dequeue operafions on a queue). The model guaranfees 
thaf fhe acfors fhaf share sfafe will be execufed in mufual exclusion, fhaf 
is, no fwo acfors fhaf share fhe same sfate will ever be executing simul- 
faneously. (This can be modeled in sfafeless dafa-flow using a single 
"sfafe" token and a non-deferminisfic merge operator @). The infroduc- 
tion of sfafe means fhat the arcs of fhe program graph no longer model 
all dependencies in fhe program; fhere are implicif dependencies via fhe 
shared sfafe. For example, consider fhe program graph fragmenf in Fig- 
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Figure 2: Hidden Dependency 


ure 1^ Suppose that actors A and B share state. If the execution of A 
occurs firsf, fhere is a hidden dependency, based on fhe sfafe, between 
A and B. Because of fhis hidden dependency, fhe resulfs of fhe A and B 
operafions depend nof only on fheir argumenfs and the object history, 
but also on the order of execufion. 


If on fhe one hand fhe persisfenf macro actors approach addresses 
the one limitation of fhe traditional data-flow model, on fhe ofher hand 
if makes fhe programming model more complicated and requires fo pro¬ 
grammers fo pay more affenfion when programming parallel applica¬ 
tions. In particular, the introduction of sfafe has one very importanf con¬ 
sequence: some programs will be deterministic, and ofhers nof. Non- 
deferminism is nof necessarily bad. There are in facf many "correcf" 
non-deferminisfic applications. Thus, it is the responsibility of fhe pro¬ 
grammer fo guarantee higher-level notions of correctness. Due to the 
additional complexity they introduce, several existing macro data-flow 
sysfems do nof supporf persisfenf acfors. 
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A notable MDF approach: the Mentat framework 

Mentat is one of the most known and used macro data-flow system ((75]|. 
It is an object-oriented parallel processing system for MIMD architec¬ 
tures developed at the University of Virginia. The computation model 
used in Mentat is a data-driven macro data-flow computation model 
based on the object-oriented paradigm. There are two primary com¬ 
ponents of Mentat: the Mentat Programming Language (MPL) and the 
Mentat run-time system. MPL is an object-oriented programming lan¬ 
guage based on C++. The computational grain of the macro data-flow 
block is the Mentat class instance, which consists of contained objects 
(local and member variables), their procedures, and a thread of control. 
Programmers are responsible for identifying those object classes that are 
of sufficient computational complexity to allow efficient parallel execu¬ 
tion. Instances of Mentat classes are used just like ordinary C++ classes. 
The data and control dependencies between Mentat class instances in¬ 
volved in invocation, communication, and S5mchronization are automat¬ 
ically detected and managed by the compiler and run-time system with¬ 
out programmer intervention. 


MPL is an extended C++ designed for developing parallel applications 
by providing parallelism encapsulation. Parallelism encapsulation takes 
two forms, intra-object encapsulation and inter-object encapsulation. In 
intra-object encapsulation of parallelism, callers of a Mentat object mem¬ 
ber function are unaware of whether the implementation of the member 
function is sequential or parallel, i.e., whether its program graph is a 
single node or a parallel graph. In inter-object encapsulation of paral¬ 
lelism, programmers of code fragments (e.g., a Mentat object member 
function) need not concern themselves with the parallel execution op¬ 
portunities between the different Mentat object member functions they 
invoke. The basic idea in the MPL is to allow the programmer to specify 
those C++ classes that are of sufficient computational complexity to war¬ 
rant parallel execution. Programmers can select which classes should be 
executed in parallel using a mentat keyword in the class definition. In- 
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stances of Mentat classes are called Mentat objects. Mentat classes are 
very similar to C++ class instance but with some minor differences (de¬ 
scribed below). The compiler generates code to construct and execute 
data dependency graphs in which the nodes are Mentat object member 
function invocations, and the arcs are the data dependencies found in the 
program. Thus, it transparently generates inter-object parallelism encap¬ 
sulation. All the communications and S 5 mchronizations are managed by 
the compiler. MPL is built around four main extensions to the C++ lan¬ 
guage. The extensions are Mentat classes, Mentat object instantiation, 
the return-to-future mechanism, and guarded select/accept statements. 

A key feature of Mentat is the transparent encapsulation of paral¬ 
lelism within and between Mentat object member function invocations. 
The hiding of whether a member function implementation is sequen¬ 
tial or parallel is called intra-object parallelism encapsulation. Similarly, 
the inter-object parallelism encapsulation consists in the exploitation of 
parallelism opportunities between Mentat object member function in¬ 
vocations in a transparent way to the programmer. Intra-object paral¬ 
lelism encapsulation and inter-object parallelism encapsulation can be 
combined. Indeed, inter-object parallelism encapsulation within a mem¬ 
ber function implementation is intra-object parallelism encapsulation as 
far as the caller of that member function is concerned. Thus, multiple 
levels of parallelism encapsulation are possible, each level hidden from 
the level above. 

Not all class objects should be Mentat objects. In particular, objects 
that do not have a sufficiently high communication ratio, i.e., whose ob¬ 
ject operations are not sufficiently computationally complex, should not 
be Mentat objects. The programmer defines a Mentat class by using the 
keyword mentat in the class definition. The programmer may further 
specify whether the class is persistent, sequential, or regular. Persistent 
and sequential objects maintain state information between member func¬ 
tion invocations, while regular objects do not. Thus, regular object mem¬ 
ber functions are pure functions. Because they are pure functions, the 
system is free to instantiate new instances of regular classes at will. Reg¬ 
ular classes may have local variables much as procedures do, and may 
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maintain state information for fhe duration of a funcfion invocafion. The 
programmer binds Menfaf variables fo persisfenf Menfaf objecfs using 
fwo reserved member funcfions for all Menfaf class objecfs: creafe() and 
bind(). The creafe() call fells fhe sysfem fo insfanfiafe a new insfance 
of fhe appropriafe class whereas fhe bind() funcfion binds Menfaf vari¬ 
ables fo an already existing insfance. The member funcfion desfroyO de- 
sfroys fhe named persisfenf Menfaf objecf. The refurn-fo-fufure func¬ 
fion (rtfO) is fhe Menfaf analog fo fhe refurn of C. Ifs purpose is fo al¬ 
low Menfaf member funcfions fo refurn a value fo fhe successor nodes 
in fhe macro dafa-flow graph in which fhe member funcfion appears. 
The selecf / accepf sfafemenfs of Menfaf is a guarded sfafemenf fhaf de¬ 
rives direcfly from fhe ADA (|lj one. Guarded sfafemenfs permif fhe pro¬ 
grammer fo specify a sef of enfry poinfs fo a monifor-like consfrucf. The 
guards are boolean expressions based on local variables and consfanfs. A 
guard is assigned fo each possible enfry poinf. If fhe guard evaluafes fo 
frue, ifs corresponding enfry poinf is a candidafe for execution. The rules 
vary for defermining which of fhe candidafes is chosen fo execufe. If is 
common fo specify in fhe language fhaf if is chosen af random. This can 
resulf in some enfry poinfs never being chosen. There are fwo f 5 rpes of 
guard-acfions supporfed by Menfaf: accepfs, fesfs, and non-enfries. Ac¬ 
cepf is similar fo fhe accepf of ADA. Tesfs are used fo fesf whefher a par- 
ficular member funcfion has any oufsfanding calls fhaf satisfy fhe guard. 
When a fesf guard-action is selecfed, no paramefers are consumed. In 
Menfaf fhere is no "else" clause as in ADA. However, using fhe priorify 
options, fhe programmer can simulafe one by specifying fhaf fhe clause 
is a non-enfry sfafemenf and giving fhe guard- sfafemenf a lower prior¬ 
ify fhan all ofher guard-sfafemenfs. Then, if none of fhe ofher guards 
evaluafes fo frue, if will be chosen. The priorify of fhe guard-sfafemenf 
defermines fhe order of evaluation of fhe guards. If can be sef eifher im- 
plicifly or explicifly. The foken priorify defermines which call wifhin a 
single guard-sfafemenf priorify level will be accepfed nexf. The foken 
priorify is fhe maximum of fhe priorifies of fhe incoming tokens. Wifhin 
a single foken priorify level, tokens are ordered by arrival time. 

To give an idea of fhe programming model in Figure we reporf 
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mentat class fibonacci_class { 
public: 

int fibonacci_class::fibonacci(int n) { 
fibonacci_class fib; 
adder_class adder; 

// if the index is 0 or 1 it returns 1 to return-to-future function 
if (n == 0 II n == 1) 
rtf (1); 

else { // otherwise it call the add method and itself recursively 
rtf(adder.add(fib.fibonacci(n - 1), fib.fibonacci(n - 2))); 

} 

return(1); 



mentat class adder_class { 
public: 

int adder_class::add(int argl, int arg2) { 

// rtf function pass the result to the successor in data-flow graph 
rtf(argl + arg2); 
return(argl + arg2); 



Figure 3: Fibonacci computation with Mentat 


a simple Mentat program. The program computes recursively the Fi¬ 
bonacci number. It is composed by two classes, the first one recursively 
computes the Fibonacci number exploiting the second one for comput¬ 
ing the sum of partial results. Clearly, in this case the efficiency is low 
because the amount of computation done by the macro actors comput¬ 
ing the mentat object adder_class is very small. 

Unfortunately, there are a number of issues and limitation that MPL 
programmers must be aware of that can lead to unpredictable program 
behavior, related both to Mentat implementation and model. Among the 
others: 

• The use of static member variables for Mentat classes is not al¬ 
lowed. Since static members are global to all instances of a class, 
they would require some form of shared memory between the in¬ 
stances of the object. 

• Mentat classes cannot have any member variables in their public 
definition. If data members were allowed in the public section. 
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users of that object would need to be able to access that data as if 
it were local. If the programmer wants the effect of public member 
variables, appropriate member functions can be defined. 

• Programmers cannot assume that pointers to instances of Mentat 
classes point to the member data for the instance. 

• Mentat classes cannot have any friend classes or functions. This 
restriction is necessary because of the independent address space 
of Mentat classes. 

• It must be possible to determine the length of all actual parame¬ 
ters of Mentat member functions, either at compile-time or at run¬ 
time. This restriction follows from the need to know how many 
bytes of the argument to send. Furthermore, each actual parameter 
of a Mentat member function must occupy a contiguous region of 
memory in order to facilitate the marshaling of arguments. 

• Mentat object member function parameter passing is call-by-value. 
All parameters are physically copied to the destination object. Sim¬ 
ilarly, return values are by-value. 

• if a Mentat member function returns a pointer, the programmer 
must explicitly delete the reference when the function is finished 
using the value. 

• semantic equivalence to the sequential program is not guaranteed 
when persistent objects are used. This is trivially true for programs 
that have select/accept statements; there are no serial equivalents. 

Summarizing Pros and Cons Data-flow model is inherently parallel, it rep¬ 
resents each computation as a graph made by operators and instructions where 
each node can be potentially executed in parallel. This model permit to program¬ 
mers to express parallel applications in a very abstract way, indeed programmers 
are not required to deal with low-level issues related to the running architecture. 
The main problem of Data-flow model is the fine-granularity of instruction that 
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prevent its exploitation in most distributed architectures and in large grid envi¬ 
ronments. This limitation led to the development of the macro data-flow model 
(MDF). The MDF model allows programmers to define codefragment in place 
of instruction as nodes in DF graph. Unfortunately, such additions impair the 
high-level abstraction, like in case of the implicit models. Hence, programmers 
have both to deal with data/application decomposition and to assure semantic 
equivalence with respect to the sequential program, especially when exploiting 
persistent actors. 

2.2.4 Low-level explicit models: MPI and OpenMP 

The low-level approaches provide to the programmers a programming 
metaphor where parallelism is represented by means of primitives in the 
form of special-purpose directives or funcfion calls. Mosf parallel primi¬ 
tives are relafed fo process S5mchronizafion, communicafion or fask par- 
fifioning. The fofal amounf of compufafional cosf for executing fhese 
primifive is considered as parallelizafion overhead. The advanfage of 
explicif parallel programming is fhe absolufe programmer confrol over 
fhe parallel execution. A very skilled parallel programmer fakes advan¬ 
fage of explicif parallelism fo produce very efficienf code. However, pro¬ 
gramming wifh explicif parallelism is offen difficulf and error prone, be¬ 
cause of fhe exfra work involved in plarming fhe fask division and S 5 m- 
chronizafion of concurrenf processes. In fhis section we reporf fwo of 
fhe main approaches fo low-level parallel computing: MPI and OpenMP. 
The former is suifable for disfribufed archifecfures whereas fhe laffer is 
appropriafe for mulficore and mulfiprocessor archifecfures. 

MPI 

MPI is a message-passing library, proposed as a sfandard by a broadly 
based committee of vendors, implementors, and programmers. MPI was 
designed for high performance on bofh massively parallel machines and 
on worksfafion clusters. The Message Passing Inferface is mean! fo pro¬ 
vide essential S5mchronizafion and communicafion funcfionalify between 
a sef of processes, mapped info differenf computer insfances, in a lan- 
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guage independent way, plus a few features that are language specific. 
The programming metaphor of MPI is based on the "process" concept. 
An MPI program consists of autonomous processes, executing their own 
code, in a Multiple Instructions, Multiple Data stream (MIMD) style, i.e. 
Multiple autonomous processors simultaneously executing different in¬ 
structions on different data. Distributed systems are generally recog¬ 
nized to be MIMD architectures. The processes communicate exploiting 
MPI communication primitives. T 5 rpically, each process executes in its 
own address space, although shared-memory implementations of MPI 
are possible. MPI does not specify the execution model for each pro¬ 
cess. A process can be sequential, or can be multi-threaded, with threads 
possibly executing concurrently. The intended interaction of MPI with 
threads is that concurrent threads be all allowed to execute MPI calls, and 
calls be reentrant; a blocking MPI call blocks only the invoking thread, 
allowing the scheduling of another thread. MPI does not provide mecha¬ 
nisms to specify the initial allocation of processes to an MPI computation 
and their binding to physical processors. MPI mapping of processes on 
PEs happens at run-time, through the agent that starts the MPI program, 
normally called mpirun or mpiexec. 

MPI primitives include, but are not limited to, point-to-point rendez¬ 
vous t 5 rpe send/receive operations, combining partial results of compu¬ 
tations (gathering and reduction operations), choosing between a Carte¬ 
sian or graph-like logical process topology, exchanging data between 
process pairs (send and receive operations), S5mchronizing nodes (bar¬ 
rier operation) as well as obtaining network-related information such 
as the number of processes in the computing session, current processor 
identity that a process is mapped to, neighboring processes accessible 
in a logical topology, and so on. Point-to-point operations come in S 5 m- 
chronous, as 5 mchronous, buffered, and ready forms in order to allow 
both relatively stronger and weaker semantics for the S5mchronization 
aspects of a rendezvous-send. Many outstanding operations are possible 
in as 5 mchronous mode, in most implementations. Figure reports the 
main classes of MPI primitives. There are two versions of the MPI stan¬ 
dard that are currently popular: version 1.2 (also called MPI-1), which 
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Figure 4: MPI-1 Routines 


emphasizes message passing and has a static run-time environment, and 
MPI-2.1 (MPI-2), which includes features such as parallel I/O, d 5 mamic 
process management and remote memory operations. Figure]^ show a 
simple Hello World MPI program. It defines two roles: masfer and slave. 
The masfer ask slaves fo process fhe "Hello word" sfring and fhen refurn 
if. The masfer evenfually prinf on screen fhe sfring received by slaves. 
The roles are specified by means of fhe MPI process id. The process num¬ 
ber 0 is fhe masfer whereas fhe ofhers are slaves. 

As shown in Figure |^MPI Hello World programmer is in charge of: 

• inifialize MPI 

• find fhe available resources and manage fhem 

• implemenf by hands a way fo differenfiafe fhe masfer and fhe slaves 

• prepare fhe dafa fhe masfer sends 

• send fhe dafa to slaves 

• make the slaves receive the data 

• implement the slave data processing 

• prepare the data the slaves send 

• make the master receive the data, collecting it and processing it 

• finalize MPI 
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furthermore, he must allocate memory buffers, manage fault(s) and dis¬ 
tribute data by hands. It is easy to understand that implement a complex 
application with MPI is a very difficult and error prone task because MPI 
programmers must manage all the aspects of the application paralleliza¬ 
tion. On one hand, it guarantees maximum programming flexibility, but 
on the other hand such a freedom is paid in terms of programming com¬ 
plexity. 

OpenMP 

Like MPI, OpenMP (Open Multi-Processing) is a specification defined 
by a group of major computer hardware and software vendors for multi¬ 
platform multiprocessing programming. It consists of a set of compiler 
directives, library routines, and environment variables that influence run¬ 
time behavior. Unlike MPI, it is mainly targeted to shared memory multi¬ 
processing. Indeed, it is used in conjunction with MPI on distributed ar¬ 
chitectures made of multicore/multiprocessor machines. OpenMP uses 
multiple, parallel threads to accomplish parallelism. A thread is a single 
sequential flow of control within a program. OpenMP uses a directive- 
based method to tell explicitly to the compiler how to distribute pro¬ 
grams across parallel threads. 

The core elements of OpenMP are the constructs for thread creation, 
workload distribution (work sharing), data environment management, 
thread S5mchronization, user level run-time routines and environment 
variables. OpenMP programmers exploit such constructs to manage all 
the aspects of application parallelization. Figure shows the classes of 
existing OpenMP language extensions. 

Even if the OpenMP approach to parallel programming has to be con¬ 
sidered as a low-level one, OpenMP code is more straightforward than 
MPI code. This is mainly due to the memory model indeed, relying on 
a shared memory model. The OpenMP application does not need to 
deal with message passing hence data are not directly split and divided 
among PEs but handled through compiler directives. 

An OpenMP program is a C++ or Portran program with OpenMP 
pragma statements/directives placed at appropriate points. The pragma 
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linclude <mpi.h> 

#include <stdio.h> 
linclude <string.h> 

Idefine BUFSIZE 128 
Idefine TAG 0 

int inain{int argc, char *argv[]) 

{ 

char idstr[32], buff[BUFSIZE]; 
int numprocs, myid, i; 

MPI_Status stat; 

/* MPI programs start with MPI_Init; all 'N' processes exist thereafter */ 
MPI_Init(&argc,&argv); 

/* find out the number of available PEs */ 

MPI_Comm_size(MPI_COMM_WORLD,Snumprocs); 

/* and this processes' rank is */ 

MPI_Comm_rank(MPI_COMM_WORLD, &myid) ; 

/* At this point, all the programs are running equivalently, the rank is 
used to distinguish the roles of the programs in the SPMD model */ 
if(myid == 0) 

{ 

/* rank 0 process sent a string to all the other processes */ 
for(i=l;i<numprocs;i++) 

{ 

sprintf(buff, "Hello %d! ", i) ; 

MPI_Send(buff, BUFSIZE, MPI_CHAR, i, TAG, MPI_COMM_WORLD); 

} 


/* rank 0 process sent a string to all the other processes */ 
for(i=l;i<numprocs;i++) 

{ 

MPI_Recv(buff, BUFSIZE, MPI_CHAR, i, TAG, MPI_COMM_WORLD, &stat); 
printf("%d: %s\n", myid, buff); 


else 

{ 

/* receive from rank 0: */ 

MPI_Recv(buff, BUFSIZE, MPI_CHAR, 0, TAG, MPI_COMM_WORLD, &stat); 
Sprintf(idstr, "Processor %d ", myid); 
strcat(buff, idstr); 

strcat(buff, "reporting for dutyXn"); 

/* send to rank 0: */ 

MPI_Send(buff, BUFSIZE, MPI_CHAR, 0, TAG, MPI_COMM_WORLD); 

} 

/* MPI Programs end with MPI Finalize */ 

MPI_Finalize(); 
return 0; 


Figure 5: Hello Word example implemented using MPI 
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Figure 6: OpenMP language extensions 


statement directs the compiler how to process the block of code that fol¬ 
lows fhe pragma. An OpenMP-enabled compiler recognizes fhe pragma 
directives and produces a parallelized execufable suifable for rurming 
on a shared-memory machine. In C/C++, an OpenMP direcfive has fhe 
general form: 

^ pragma omp directive — name [clause,...] newline 

The #pragma omp direcfive fags a block for parallel or various fypes 
of work sharing execufion, variable scoping and S5mchronizafion consid- 
erafions. One or more clauses are opfional and may be in any order. The 
clauses are used fo explicifly define fhe scoping of enclosed variables. In 
OpenMP fhere are two main constructs: 

• A parallel region is a block of code that will be executed by multiple 
threads. This is the fundamental parallel construct. 

• A work-sharing construct divides the execution of the enclosed 
code region among the members of the team that encounter it. Work¬ 
sharing constructs do not launch new threads. These constructs 
are identified by DO/FOR, SECTIONS and WORKSHARE (Eortran 
only) directives. 
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int main (int argc, char *argv[]) { 
int nthreads, tid, i, chunk; 
float a[N], b[N], c[N]; 

/* Some initializations */ 

for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; 
chunk = CHUNKSIZE; 

#pragma omp parallel shared(a,b,c,nthreads,chunk) private(i,tid) 
{ 

tid = omp_get_thread_num(); 
if (tid — 0) { 

nthreads = omp_get_num_threads{); 

printf (’’Number of threads = %d\n”, nthreads); 

} 

printf ( "Thread %d starting. ..\n’', tid) ; 

Ipragma omp for schedule(dynamic,chunk) 
for (i=0; i<N; i++) { 

c [i] = a [i] + b [i] ; 

printf("Thread %d: c[%d]= %fXn",tid,i,c[i]); 

} 

} /* end of parallel section */ 

} 


Figure 7: Factorial example in OpenMP 


Since OpenMP is a shared memory programming model, most vari¬ 
ables in OpenMP code are visible to all threads by default. However, 
sometimes private variables are necessary to avoid a race condition and 
there is a need to pass values between the sequential part and the paral¬ 
lel region. Another important issue is the synchronization and schedul¬ 
ing of fhe threads. These are managed through clauses appended to the 
OpenMP directive. Thus, the different t5rpes of clauses are Dafa Scoping, 
S5mchronization and Scheduling clauses. 

In Figure we reporf an OpenMP example program. The example 
uses fwo pragma direcfives. The oufer #pragma omp parallel fags a block 
for parallel execution. The sharedO clause specifies common variables, 
and private 0 specifies fhe variables resfricfed fo exclusive use by a pro¬ 
cess. The inner #pragma omp for schedule directive specifies disfribu- 
tion across fhreads. The fhreads share fhe variables a, b, c and chunk; 
the iteration variable i is private in each thread. The expression tells the 
compiler to perform parallel execution of fhe for-loop and fo splif fhe 
iferafion space info blocks of size chunk. 
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The current version of OpenMP presents some issues, some related 
to the implementation and others related to the model. For instance a 
reliable error handling, fine-grained mechanisms for controlling thread- 
processor mapping or S5mchronization among a subset of threads. The 
model related issues, clearly more difficult to overcome include inef¬ 
ficient parallelism exploitation in distributed-memory platforms and a 
limited scalability that actually depends by memory architecture. 


Summarizing Pros and Cons Low-level approaches allow programmers to 
control all the aspects of parallel applications and their execution. Exploiting 
low-level approaches skilled programmers can implement very efficient parallel 
applications. The freedom and efficiency allowed by the model are paid in terms 
of expressiveness and ease of use. Indeed, programmers have to manage "by 
hand" all the issues related to data and program decomposition, fault tolerance, 
load balancing and communications. 

2 . 2.5 Other notable approaches 

Other two noteworthy explicit parallel approaches are Cilk and High 
Performance Fortran. 

The first one is quite similar to OpenMP, indeed it consists in an en¬ 
riched version of C language, it requires that the computing resources 
share the main memory hence can be used for programming parallel 
applications rurming in multiprocessor machines but not in distributed 
architecture like clusters. It enriches GNU C with a few Cilk-specific 
keywords. Using them programmers expose the parallelism identifying 
elements that can safely be executed in parallel. Using such information 
the run-time environment, in particular the scheduler, decides during ex¬ 
ecution how to distribute the work among processors. The first Cilk key¬ 
word is cilk, which identifies a function written in Cilk. Since Cilk pro¬ 
cedures can call C procedures directly, but C procedures cannot directly 
call or spawn Cilk procedures, this ke5rword is needed to distinguish Cilk 
code from C code. Other keywords are: spawn, S5mc, inlet and abort. The 
first two ke5rwords are all Cilk programmers have to use to start using 
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the parallel features of Cilk: spawn indicafes fhaf fhe procedure call if 
modifies can safely operafe in parallel wifh ofher execufing code. Nofe 
fhaf from fhe poinf of view of fhe scheduler if is nof mandafory fo run 
fhis procedure in parallel; fhe ke5rword only inform fhe scheduler fhaf if 
can run fhe procedure in parallel. S5mc indicafes fhaf execufion of fhe cur- 
renf procedure carmof proceed unfil all previously spawned procedures 
have complefed and refumed fheir resulfs fo fhe parenf frame. The fwo 
remaining Cilk ke5rwords are slighfly more advanced, and concern fhe 
use of inlefs. T5rpically, when a Cilk procedure is spawned, if can only 
refurn ifs resulfs fo fhe parenf procedure by puffing fhose resulfs in a 
variable in fhe parenf's frame, as we assigned fhe resulfs of our spawned 
procedure calls in fhe example fo x and y. The alfemafive is fo use an 
inlet. An inlet is a function internal to a Cilk procedure that handles the 
results of a spawned procedure call as fhey refum. One major reason fo 
use inlefs is fhaf all fhe inlefs of a procedure are guaranfeed fo operafe 
atomically wifh regards fo each ofher and fo fhe parenf procedure, fhus 
avoiding fhe bugs fhaf could occur if fhe multiple refurning procedures 
fried fo updafe fhe same variables in fhe parenf frame af fhe same fime. 
The abort ke5rword can only be used inside an inlet; it tells the sched¬ 
uler that any other procedures that have been spawned off by fhe parenf 
procedure can safely be aborfed. 

High Performance Forfran is an extension of Forfran 90 defined by 
fhe high performance forfran forum wifh consfrucfs fhaf supporf dafa- 
parallel compufafions. If consisfs in a porfable language for dafa-parallel 
compufafions. HPF uses a dafa parallel model of compufafion fo sup¬ 
porf spreading fhe work of a single array compufafion over multiple pro¬ 
cessors. This allows efficienf implemenfafion on bofh SIMD and MIMD 
sfyle archifecfures. If provides a number of basic dafa parallel functions 
as builf-in array operators and infrinsic funcfions. If also provides con¬ 
sfrucfs, such as fhe where and fhe forall, which assisf in programming 
more complex dafa parallel funcfions. The simplesf dafa parallel opera¬ 
tions are fhe elementwise operations. For any base operafion on a dafa 
fype, programmers can extend fhaf operafion fo an array operafion. For 
binary (and higher degree) operafions, fhe arrays musf have fhe same 
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shape. The result of the operation is another array of that shape, in which 
the elements are defined by the elementwise extension of the base oper¬ 
ation. A more advanced set of operations operate on an entire array to 
produce a single answer, they implement a behavior generally known 
as reduction. Reduction can be defined for any associative, binary op¬ 
eration that produces a result of the same element type by successively 
accumulating the results of applying that operation to elements of the 
array. Commonly used operations include arithmetic operators like ad¬ 
dition, multiplication, maximum, and minimum and boolean operators. 
As an example, HPF programmers can define reduction with addition, 
usually called sum reduction, over any array whose element type can be 
added. 


2.2.6 Structured approach 

Highly abstract approaches and low-level approaches represent the two 
extremes in parallel programming models. The formers completely au¬ 
tomate the aspects of parallelization, namely do not ask programmers (at 
least in their "pure" version) to give any information about application, 
like data distribution and S5mchronization, communication mechanisms, 
executing environment or code sequences to run in parallel. The latter, 
opposite, approaches do not automate anything and ask programmers to 
deal, almost entirely, with the application parallelization aspects. 

As we outlined in previous sections, several researchers have tried 
to address the limitation of these approaches enriching them with addi¬ 
tional features. Some other work was done trying to conceive alterna¬ 
tive models. In particular, since the nineties, several research groups 
have proposed the structured parallel programming environments{SFFE). 
Since the structured parallel programming model was conceived, several 
works have been done about it, also from a foundational point of view 
< [20) , (HD, ([soil . Programming environments relying on this paradigm 
(i.e. dlOlU ask programmers to explicitly deal with the qualitative as¬ 
pects of parallelism exploitation, namely the application structure and 
problem decomposition strategies. All the low-level parallelism exploita- 
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tion related aspects like communication, S5mchronization, mapping and 
scheduling are managed by compiler tools and run-time support. 

The structured way is driven by two observations: that there are some 
things people do better than compilers, and that there are some things 
that compilers do better than people. Rather than have either do the 
complete job, it exploits the comparative advantages of each. Indeed the 
management of fens fo fhousands of asynchronous fasks, where timing- 
dependenf errors are quife common, is beyond fhe capacify of mosf pro¬ 
grammers whereas compilers are very good af ensuring fhaf evenfs hap¬ 
pen in the right order and can more readily and correctly manage com¬ 
munication and S 5 mchronization than programmers. On the other hand, 
data decomposition strategies and computational grain can be successful 
managed by programmers buf nof efficienfly by compilers. 

The environmenfs following fhis way are fhose based on fhe algorifh- 
mic skeleton concepf. A skelefon, is a known and widely used paffem of 
parallelism exploifation originally conceived by Cole (l50l and lafer on by 
differenf research groups fo design high-performance sfrucfured parallel 
programming environmenfs. 

Basically, sfrucfured parallel programming sysfems allow a parallel 
applicafion fo be coded by properly composing a sef of basic parallel 
skeletons. These basic skelefons usually include skeletons modeling em¬ 
barrassingly parallel compufafions (farms), compufafions sfrucfured in 
sfages (pipelines) as well as common dafa parallel compufafion patterns 
(map/forall, reduce, scan). Each skelefon is paramefric; in parficular, 
if accepfs as a parameter fhe kind of compufafion fo be performed ac¬ 
cording to parallelism exploitation pattern it models. As an example, a 
farm skelefon fakes as a parameter fhe worker, i.e. fhe compufafion fo be 
performed on fhe single inpuf fask (dafa ifem). As a furfher example, a 
pipeline fakes as paramefers fhe pipeline sfages. Such parameters may 
be eifher paramefers modeling sequential portions of code (sequential 
skeletons) or even other skeletons, in turn. Therefore, a farm skelefon 
may fake as a worker a two sfage pipeline. The composifion of fhe fwo 
expresses embarrassingly parallel compufafions where each inpuf fask 
(dafa ifem) is processed by fwo sfages. Parallelism is exploited bofh by 


47 


using different resources to compute independent input tasks and by us¬ 
ing different resources to compute the first and the second stage onto a 
single input task. 

A skeleton (in its original formulation) is formally an higher order 
function taking one or more other skeletons or portions of sequential 
code as parameters, and modeling a parallel computation out of them. 
Cole's skeletons represent parallelism exploitation patterns that can be 
used (instanced) to model common parallel applications. Later, different 
authors figure out that skeletons can be used as constructs of an explicitly 
parallel programming language, actually as the only way to express par¬ 
allel computations in these languages dSOllMt . Recently, the skeleton con¬ 
cept evolved, and became the coordination layer of structured parallel 
programming environments ( (l29ll32l[ll3H . In any case, a skeleton can be 
considered as an abstraction modeling a common, reusable parallelism 
exploitation pattern. Skeletons can be provided to the programmer either 
as language constructs (l29ll30H32ll or as libraries (IT2tl55ll62ll88t . Usually, 
the set of skeletons includes both data-parallel and task parallel patterns. 

Traditional skeleton approaches 

From the nineties, several research groups proposed or currently propose 
programming environments supporting parallel computations based on 
the algorithmic skeleton concept. They are implemented as frameworks, 
languages or libraries. Among the others, we mention Kuchen's C++ 
MPI skeleton library ( |55t , Serot's SKiPPER environment, P^L, Lithium, 
a first version of muskel and JJPF. In particular, the last one, JJPF, rep¬ 
resents our approach to traditional SPPE. In the rest of this section we 
present a more detailed description about the programming model of 
P^L, muskel and JJPF to describe the "concept behind" SPPE models. 
We developed this last one, whereas all the other skeleton environments 
presented in this section have been developed by the Parallel and Dis¬ 
tributed Architecture Group, part of the Department of Computer Sci¬ 
ence at University of Pisa. This group has a deep background on skeleton 
environment, indeed the group began to work in this field from the very 
begirming the skeleton model were conceived. We collaborated with sev- 
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eral researchers belonging to this group, also for the conception and the 
design of fhe resulfs presenfed in fhis fhesis. 

P^L is a high-level sfrucfured explicifly parallel language developed in 
the nineties dSH . Using P^L parallelism can be expressed only by means 
of a resfricfed sef of parallel consfrucfs each corresponding fo a specific 
parallel form. Sequential parts are expressed by using an existing lan¬ 
guage also called the host sequential language of P^L. Being a SPPE 
ifs consfrucfs can be hierarchically composed fo express more complex 
parallel forms. This composifional properfy relies on fhe semanfics asso- 
ciafed wifh fhe various P^L consfrucfs and fheir composifions. In facf, 
each of fhem can be fhoughf of as a dafa-flow module. In P^L each mod¬ 
ule compufes in parallel or sequentially a function on a given sfream of 
inpuf dafa and produces an oufpuf sfream of resulfs. The lengfhs of bofh 
fhe sfreams are idenfical and fhe ordering is preserved, i.e. 

[zui,..., fn„] —>■ M —>■ [outi ,..., outn] 

where M is fhe dafa-flow module corresponding fo a generic P^L con- 
sfrucf [ini,..., m„] is fhe inpuf sfream, [outi,..., outn] is fhe oufpuf sfream, 
n is fhe lengfh of bofh fhe sfreams and every oufpuf dafa ifem outi is ob- 
fained by applying fhe function compufed by M on fhe inpuf dafa ifem 
iui. The f 5 rpes of fhe inpuf and fhe oufpuf inferface of each P^L consfrucf 
i.e. fhe fypes of every and every outi have fo be declared sfafically. 
Acfually fhe compiler performs type checking on these interfaces when 
the P^L constructs are to be composed. Another feature of P^L is ifs 
inferface wifh the host sequential language. The interface has been de¬ 
signed to make easier portability between different host languages. In 
fact, sequential parts are completely encapsulated into the constructs of 
P^L. Paramefer passing between P^L consfrucfs are handled by linguis¬ 
tic consfrucfs fhaf are exfernal fo fhe specific hosf sequential language 
while fhe dafa f 5 rpes fhaf can be used fo define fhe inferface of fhe P^L 
consfrucfs are a fixed subsef of fhose usually available in fhe mosf com¬ 
mon languages. The firsf P^L compiler adopfed as hosf sequenfial lan¬ 
guage C and C++. The consfrucfs included since fhe firsf P^L compiler 
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• The farm construct which models processor farm parallelism. In 
this form of parallelism a set of identical workers execute in parallel 
the independent tasks that come from an input stream and produce 
an output stream of results. 

• The map construct which models data parallel computations. In 
this form of parallelism each input data item from an input stream 
is decomposed into a set of partitions and assigned to identical and 
parallel workers. The workers do not need to exchange data to per¬ 
form their data parallel computations. The results produced by the 
workers are recomposed to make up a new data item of an output 
stream of results. 

• The pipe construct which models pipeline parallelism. In this form 
of parallelism a set of stages execute serially over a stream of input 
data producing an output stream of results. 

• The loop construct which models computations where for each in¬ 
put data item a loop body has to be iteratively executed until a 
given condition is reached and an output data item is produced. 

• The sequential construct which corresponds to a sequential pro¬ 
cess that for each data item coming from an input stream produces 
a new data item of an output stream 

The sequential constructs constitute the leaves of the hierarchical compo¬ 
sition because the computations performed by them have to be expressed 
in terms of the host sequential language. 

muskel (jSSJ is a full Java framework, providing programmers with 
structured ways of expressing parallel programs. The muskel environ¬ 
ment represents a sensible evolution of the Lithium one (IT2ll . It inher¬ 
its from Lithium the normal form and macro data-flow (1571 llOlt im¬ 
plementation techniques as well as the general structure of the run-time 
support. 


50 



Normalization consists in transforming the original skeleton tree (or 
composition) into a program that is basically a task farm wifh sequenfial 
workers Such optimization basically subsfifufe skeleton subfrees 
by skelefon subfrees providing a better performance and efficiency in 
fhe fargef machine resource usage fhan fhe original skelefon free. Previ¬ 
ous resulfs demonsfrafed fhaf full sfream parallel skelefon subfrees can 
be collapsed fo a single farm skelefon wifh a (possibly huge) sequenfial 
worker leading fo a service fime which is equal or even better fhaf fhe 
service fime of fhe uncollapsed skelefon free dSOl . 

The muskel macro dafa-flow run-fime supporf consisfs in deriving 
a graph of macro dafa-flow blocks from skelefon frees and dispafching 
fhem fo compufafional resources rurming macro-acfors. 

muskel adds fo Lifhium a limited form of resource discovery and 
faulf tolerance feafures as well as fhe whole Application Manager concepf. 

The Application Manager{AM) is an entify fhat fakes care of assuring 
thaf fhe applicafion non-funcfional requirement were satisfied. The re- 
quiremenfs are specified by programmers in a performance contracf. The 
AM acfively observes fhe applicafion behavior and in case of faulfs or 
performance confracf violations it reacts aiming to fix fhe problem, as 
an example, in case of a compufafional resource faulf if recruifs a new 
resource in fhe compufafion. 

Using muskel a programmer can implemenf parallel programs fhaf 
match the task farm or fhe pipeline parallelism exploifation patterns as 
well as arbifrary composition of fhe fwo. Despite fhe limited amounf of 
patterns supported, however, a large range of applicafions can be pro¬ 
grammed, for insfance all embarrassingly parallel applicafions, parame- 
fer sweeping applicafions and mulfisfage applicafions. 

A fask farm compufafion can be defined jusf using a Farm objecf. 
The Farm consfrucfor takes a parameter representing the computation 
performed by fhe farm workers. This compufafion can be eifher a se¬ 
quential computation or another parallelism exploitation pattern (an¬ 
other Farm or a Pipeline one). A pipeline computation can be defined 
using a Pipeline objecf. The Pipeline consfrucfor fakes fwo parameters 
fhaf can eifher be sequenfial compufafion objecfs or in torn parallel ex- 
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ploitation patterns. Pipelines with more stages can be obtained compos¬ 
ing several Pipeline objects. Then the programmer has to add an Appli¬ 
cation Manager to the application code, and he must also specify the per¬ 
formance confracf he prefends fo be respecfed on fhe fargef archifecfure. 
This is done insfanfiafing an application manager and specifying a per¬ 
formance confracf. muskel supporfs two different kinds of confracfs. 
The firsf one requires a consfanf parallelism degree, fhaf is, if requires 
fhaf a consfanf number of processing elemenfs are dedicafed fo fhe par¬ 
allel execution of our parallel program. The second one requires fhaf a 
given fhroughpuf is mainfained in ferms of fask processed per unif fime. 
Bofh of fhese kinds of confracfs can be specified before fhe compufafion 
of fhe parallel muskel program acfually sfarfs and can be changed dur¬ 
ing fhe program execution. The managemenf of fhe parallel compufafion 
in such a way fhaf fhe confracfs are safisfied is complefely handled by an 
independenf execution flow. Therefore, fhe submission of a new perfor¬ 
mance confracf fo fhe application manager immediafely friggers all fhose 
(possibly additional) activities needed fo satisfy fhe confracf. The possi- 
bilify fo change fhe performance confracfs during fhe execution of fhe 
parallel applications allows fhe programmer fo implemenf some kind of 
applicafion dependenf dynamic execufion sfrafegy. Once fhe program 
has been specified along wifh ifs performance confracf fhe programmer 
musf supply fhe lisf/sfream of fasks fo be compufed. When all fhe el¬ 
emenfs belonging fo fhe lisf/sfream have been processed, fhe parallel 
execufion of fhe program is ferminafed and fhe relative resulfs can be 
fefched. 

During fhe compufafion of fhe parallel program fhe muskel run- 
fime aufomafically discovers available processing elemenfs. In case fhere 
are no enough resources fo satisfy fhe confracf, an error is signaled fo fhe 
programmer. 

As we sfafed before, in case of faulfs fhe Applicafion Manager recruifs 
new resources among fhe available ones fo subsfifufe fhe faulfy one. In 
case fhe applicafion manager recognizes fhaf fhe performance confracf 
specified by fhe programmer carmof be safisfied, if raises an Excepfion. 
Being any fask fo be compufed a fireable macro dafa flow insfrucfion, if is 
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completely independent of any other task needed to compute the parallel 
application. Therefore, if can be scheduled on any one of fhe available 
resources. However, fhe normal form concepf implemenfed in muskel, 
only generafes fully independenf macro dafa flow insfrucfions. Thai is, 
no resulf of an insfrucfion is needed fo compufe anofher insfrucfion. In 
fhis case, mosf of fhe scheduling problems we jusf menfioned disappear. 

JJPF is a parallel programming framework builf on fop of plain Java 
fhaf can run sfream parallel applications on several parallel / distributed 
architectures ranging from fightly coupled worksfafion clusfers fo generic 
worksfafion nefworks and grids. In a sense, JJPF represenfs our approach 
to old-fashioned sfrucfured parallel programming environmenfs. If di- 
recfly inherifs from fhe early versions of Lifhium and muskel (1121 . Bofh 
Lithium and muskel exploit plain RMI Java technology to distribute 
computations across nodes, and rely on NFS (the network file sysfem) fo 
disfribufe fhe applicafion code fo fhe remofe processing elemenfs. JJPF, 
insfead, is fully implemenfed on fop of JINI and Java and relies on fhe 
Jini Exfensible Remofe Invocafion (JERI) mechanism fo disfribufe code 
across fhe remofe processing nodes involved in sfream parallel applica¬ 
fion compufafion. JJPE exploifs fhe sfream parallel sfrucfure of fhe appli¬ 
cafion in such a way fhaf several disfincf goals can be achieved: 

• load balancing is achieved across fhe compufing elemenfs parficipaf- 
ing in fhe compufafion 

• processing elemenfs available fo parficipafe fo fhe compufafion of 
sfream parallel application are automatically discovered and recruited 
exploiting standard Jini mechanisms 

• faulty processing elements are automatically substituted by fresh ones 
(if any) in a seamless and aufomafic way. Therefore, fhe sfream 
parallel applications computations resist to both node and network 
faults. Programmers do not need to add a single line of code in his 
applicafion fo deal wifh faulfy nodes/nefwork, nor if has fo fake 
any ofher kind of acfion fo gef advanfage of fhis feafure. 
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JJPF has been tested using both S5mthetic and real applications, on both 
production workstation networks and on clusters, with very nice and 
encouraging results. JJPF has been designed to provide programmers 
with an environment supporting the execution of stream parallel appli¬ 
cations on a network of worksfafions, exploifing plain Java fechnology. 
Overall JJPF provides a disfribufed server providing a sfream parallel 
applicafion compufafion service. Programmers musf wrife fheir appli- 
cafions in such a way fhey jusf exploif an arbifrary composifion of fask 
farm and pipeline patfems. Task farm only applicafions are direcfly ex- 
ecufed by fhe disfribufed server, while applicafions exploifing compo¬ 
sifion of fask farm and pipeline patfems are firsf processed fo gef fheir 
normal form. A disfribufed environmenf fhaf exploifs fask parallel com- 
pufafions, permifs fo implemenf differenf applicafions in really differenf 
applicative and hardware contexts. JJPF is based on a master-worker 
architecture. JJPF defines fwo entifies: "clienf", fhaf is fhe applicafion 
code (fhe masfer), and "service", fhat consisfs in disfribufed server in- 
sfances (fhe workers) fhaf acfually compufe resulfs ouf of input task data 
to execute client program. Figure|^sketches the structure of fhe fwo com- 
ponenfs. The clienf componenf basically recruifs available services and 
forks a confrol fhread for each one of fhem. The confrol fhread, in fum, 
fefches uncompufed fask ifems from fhe fask vecfor, delivers fhem fo fhe 
remofe service and refrieves fhe compufed resulfs, storing fhem fo fhe 
resulf reposifory. Low-level acfivifies, like resource recruifing, program 
deploymenf and dafa fransfer are performed direcfly by fhe framework 
exploifing fhe JINI fechnology (|D. The key concepf in JJPF is fhaf ser¬ 
vice discovery is aufomafically performed in fhe clienf run time supporf. 
Nof a single line of code dealing wifh service discovery or recruifing is fo 
be provided by application programmers. JJPF achieves automatic load 
balancing among the recruited services, due to the scheduling adopted 
in the control threads managing the remote services. Furthermore, it 
handles faults in service nodes automatically taking care of fhe fasks as¬ 
signed fo a service node in such a way fhat in case the node does not 
respond any more they can be rescheduled to other service nodes. This 
is only possible because of fhe kind of parallel applicafions fhat are sup- 
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Figure 8: Simplified state diagram for the generic JJPF client (left) and ser¬ 
vice (right) 


ported in JJPF, that is stream parallel computations. In this case, there 
are natural descheduling points that can be chosen to restart the computa¬ 
tion of one of the input tasks, in case of failure of a service node. JJPF 
has demonstrated good scalability both in embarrassingly parallel appli¬ 
cation and in more "problematic" applications. 


2.3 Open issues in structured approaches 

Despite being around since long time and despite the progress made in 
skeletal system design and implementation, the skeleton systems did not 
take off as expected. Nowadays, the skeleton system usage is actually re¬ 
stricted to small communities grown around the teams that develop the 
skeleton systems. Cole focused very well the problem in his manifesto 
dS). Here he stated four principles that have to be tackled in skeletal 
systems to make them effective and successful: 

I) Propagate the concept with minimal conceptual disruption It means 
that skeletons must be provided within existing programming environ¬ 
ments without actually requiring the programmers to learn entirely new 
programming languages. In order to make them widely used by practi¬ 
tioners they should not require further conceptual baggage. 
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II) Integrate ad-hoc parallelism Many parallel applications are not ob¬ 
viously expressible as instances of skeletons. Some have phases that re¬ 
quire the use of less sfrucfured inferacfion primitives. For example. Can¬ 
non's well-known mafrix multiplication algorifhm (l90t invokes an initial 
sfep in which mafrices are skewed across processes in a manner which 
is nof efficienfly expressible in many skelefal sysfems. If is unrealistic fo 
assume fhaf skeletons can provide all fhe parallelism we need. We musf 
consfrucf our sysfems fo allow fhe infegrafion of skelefal and ad-hoc par¬ 
allelism in a well-defined way. 

III) Accommodate diversity All the existing skeleton systems have a 
common core of simple skeletons and a variefy of more exofic forms. 
When described informally, fhe core operafions are sfraighfforward. In¬ 
stead, precise specificafion reveals variations in semantics fhaf reflecf fhe 
ways skeletons are applied in real algorifhms. The resulf is fhat some 
algorifhms, which intuifively seem fo represenf an insfance of a skelefon, 
cannof be expressed in cerfain sysfems because of consfrainfs imposed by 
fhe specification. Hence, skeletal systems should provide mechanisms to 
specialize skeletons, in all those cases where specialization does not rad¬ 
ically change the nature of fhe skelefon, and consequenfly fhe nafure of 
the implementation. 

IV) Show the pay-back A new technology will only gain acceptance 
if if can be demonsfrafed fhat adoption offers some improvemenf over 
fhe sfafus quo. The sfrucfural knowledge embedded in skeletons should 
allow optimization wifhin and across uses fhaf would nof be realistically 
achievable by hand, i.e. demonsfrafe fhaf fhe efforf required fo adopf a 
skelefal sysfem is immediately rewarded by some kind of concrefe re- 
sulfs: shorter design and implemenfafion time of applications, increased 
efficiency, increased machine independence of fhe applicafion code, efc. 

The second and fhe third points are specifically technical whereas the 
first and the last one are actually a kind of "adverfising" ones, in a sense. 
All fhese poinfs, however, have impacfs on bofh fhe way fhe skelefon 


56 


systems are designed and on the way they are implemented. The Cole's 
analysis is not the only one, (IT9t extends it adding some other features 
a skeleton environment have to address to be suitable for fhe compufa- 
fional grids. In particular, fhe aufhors presenf fhree more requiremenfs 
for Skelefal sysfems: 

V) Support code reuse that is allow programmers to reuse with mini¬ 
mal effort existing sequential code; 

VI) Handle heterogeneity i.e. implement skeletons in such a way skele¬ 
ton programs can be run on clusters/networks/grids hosting heteroge¬ 
neous computing resources (different processors, different operating sys¬ 
tems, different memory/disk configurations, etc.); 

VII) Handle dynamicity i.e. implement in the skeleton support mech¬ 
anisms and policies suitable to handle t 5 q)ical d 5 mamic situations, such 
as those arising when non-dedicated processing elements are used (e.g. 
peaks of load fhaf impair load balancing sfrafegies) or from sudden un- 
availabilify of processing elemenfs (e.g. nefwork faulfs, node reboof). 

Summarizing, fhe nexf generafion of Skelefal Sysfems, fhaf drawing 
a parallel wifh web programming model we can refer as "Skelefons 2.0", 
have fo infegrafe ad-hoc parallelism and provide mechanisms fo special¬ 
ize skelefons in order fo express cusfomized form of parallel exploifa- 
fion. They have fo supporf code reuse, handle heferogeneify and dy- 
namicify in order fo be exploifed in grid environmenfs. Moreover, such 
feafures musf be provided wifh minimal concepfual disruption, hence 
wifhouf requiring fhe programmers fo learn entirely new programming 
languages or environmenfs buf infegrafing "Skelefons 2.0" principles in¬ 
side fhe exisfing programming fools, possibly wifhouf changing fheir 
programming absfracfion. 

Some Skelefal sysfems have addressed fhe "Skelefons 2.0" principles 
fo differenf degrees in differenf combinafions. Nexf secfion reporfs some 
of fhe mosf nofable among fhese sysfems. 
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2.3.1 Attempts to address issues 

In its "manifesto" paper Murray Cole, together with the check-list of is¬ 
sues fhaf nexf generafion of skeleton system should address, skefches fhe 
eSkel library dSTt . eSkel consisfs in Cole's aftempi to address fhe issues 
he presenf in his "manifesto" paper. More in defail, eSkel is a library 
of C funcfions and fype definifions fhaf exfends fhe sfandard C binding 
fo MPI wifh skelefal operafions. Ifs underlying concepfual model is fhe 
SPMD disfribufed memory model, inherited from MPI, and ifs opera¬ 
fions musf be invoked from wifhin a program fhaf has already inifial- 
ized an MPI environmenf. eSkel provides programmers wifh some lan¬ 
guage primitives performing complex operafions fhaf can be infegrafed 
wifh fhe fradifional MPI funcfions. eSkel implemenfs skelefons as col¬ 
lective MPI operafions. In ISTIl aufhors describe how fhe manifesto 
issues are addressed in eSkel. eSkel also provides some code reuse facil- 
ifies (check-lisf poinf V) as mosf C and C++ code can simply be adapfed 
in eSkel programs. In eSkel heterogeneous archifecfures are supporfed 

(VI) fhrough fhe usage of MPI, much in fhe sense heterogeneous archi¬ 
fecfures are supporfed fhrough fhe usage of Java in muskel. However, 
currenf implemenfafion of eSkel does nof supporf cusfom, programmer 
defined, MPI dafa f 5 rpes in fhe communicafion primitives, fhaf acfually 
use MPLINT dafa buffers, and fherefore heterogeneous archifecfures can 
be fargefed using proper MPI implemenfafions jusf when all fhe nodes 
have fhe same type of processors. No supporf for d 3 mamicify handling 

(VII) is provided in eSkel, however. 

Some ofher groups involved in sfrucfured parallel programming re¬ 
search, developed programming sysfems fhaf parfially address fhe is¬ 
sues above presenfed. Schaeffer and his group af fhe Universify of Al- 
berfa fhaf implemenfed a sysfem were programmers can inserf new par¬ 
allelism exploifafion pafferns in fhe sysfem (l38t . Kuchen Muesli (l89t 
is basically a C++ library builf on fop of MPI providing sfream paral¬ 
lel skelefons, dafa parallel objecfs and dafa parallel operafions as C++ 
femplafe classes. The programming interface is definitely very good, 
as fhe full power of objecf orienfed paradigm along wifh templates is 
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exploited to provide Muesli programmers with user-friendly skeletons, 
and consequently C++ programmers can develop parallel applications 
very rapidly. In particular. Muesli does not require any MPI specific 
knowledge/acfion fo wrife a skeleton program. Therefore, poinf (I) is 
very well addressed here. Poinfs (II) and (III) are addressed providing 
fhe programmer wifh a full sef of (dafa parallel) operations fhaf can be 
freely combined. The payback (IV) is mainly relafed fo fhe OO fech- 
niques exploited fo provide skeletons. Code reuse (V) is supporfed as 
if is supporfed in eSkel, as programmers can use C++/C code fo build 
fheir own skelefons as well as sequential code fo be used in fhe skele- 
fons. Even in fhis case fhere is limifed supporf fo heferogeneify (VI): fhe 
MPI code in fhe Skeleton library direcfly uses MPLBYTE buffers fo im- 
plemenf Muesli communications, and fherefore MPI libraries supporfing 
heferogeneous archifecfures may be used jusf in case fhe nodes sporf fhe 
same kind of processor and fhe same C/C++ compiler fool-sef. Dynam- 
icify handling (VII) is nof supporfed af all in Muesli. 

Gorlafch's and ifs research group presenfed a grid programming en- 
vironmenf HOC (l73t , which provides suifable ways of developing com- 
ponenf based grid applications exploiting classical skeleton componenfs. 
The implemenfafion exploifs Web Services fechnology. Overall, fhe HOC 
programming environmenf addressed principles (I) and (IV). Poinfs (II) 
and (III) rely on fhe possibilify given fo programmers fo inserf / create 
new HOCs in fhe repository. Poinf (VI) is handled via Web Services. This 
fechnology is inherenfly mulfiplafform, and fherefore heferogeneous far- 
gef archifecfures can be easily used fo run HOC programs. Poinf (V) is 
guaranfeed as sequenfial code can easily (modulus fhe facf some XML 
code is needed, acfually) be wrapped in Web Services. However, no sup¬ 
porf fo (VII) is included in fhe currenf HOC version. 

2.4 Our efforts in designing "Skeletons 2.0" 
systems 

Even fhough Cole and ofher research groups, focused on skeleton sys- 
fem, designed and developed skelefon systems fhaf own some of fhe 
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features required to be a next generation skeleton system, the research 
for addressing the presented issues is just started. In fact, up to now 
tools and model that are generally recognized as the best solutions for 
addressing the issues presented in dSTl and in (1191 simply do not exist. 
In the Chapters 1^1^ and 1^ we present some models and the concerning 
tools that we designed and developed in order to contribute to research 
for next generation skeleton systems. 

More in detail, in Chapter we propose a macro data-flow based ap¬ 
proach designed supporting the integration of unstructured form of par¬ 
allelization in skeleton systems, hence addressing the issue number II. 
To validate the approach we modified a skeleton system that in its orig¬ 
inal form does not deal with unstructured parallelism: muskel. We 
extended muskel, in collaboration with the research staff that develop 
it, to integrate it with a methodology that can be used to implement 
mixed parallel programming environments providing the programmer 
with both structured and unstructured ways of expressing parallelism. 
The methodology is based on data-flow. Structured parallel exploitation 
patterns are implemented translating them into data-flow graphs exe¬ 
cuted by a distributed macro data-flow interpreter. Unstructured paral¬ 
lelism exploitation can be achieved by explicitly programming data-flow 
(sub)graphs. The modified muskel provides suitable ways to interact 
with the data-flow graphs derived from structured pattern compilation 
in such a way that mixed structured and unstructured parallelism ex¬ 
ploitation patterns can be used within the same application. Two mech¬ 
anisms provided to the muskel programmers for unstructured paral¬ 
lelism exploitation. First, we provide primitives that allow accessing the 
fundamental features of the data-flow graph generated out of the com¬ 
pilation of a skeleton program. Namely, methods to deliver data to and 
retrieve data from data-flow graph. We provide to programmers the abil¬ 
ity to instantiate a new graph in the task pool by providing the input task 
token and to redirect the output token of the graph to an arbitrary data¬ 
flow instruction in the pool. Second, we provide the programmer with 
direct access to the definition of data-flow graphs, in such a way he can 
describe his particular parallelism exploitation patterns that cannot be ef- 
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ficiently implemented with the available skeletons. The two mechanisms 
can be jointly used to program all those parts of the application that can¬ 
not be easily and efficiently implementing using the skeletons subsys¬ 
tem. Unfortunately, this approach is not free from shorfcomings In facf 
exploifing unsfrucfured parallelism inferacfing direcfly wifh dafa-flow 
graph requires fo programmers fo reason in ferms of program-blocks in- 
sfead of a monolifhic program. Hence, af a firsf sighf fhis approach may 
look like fhe ones presenf in fhe ofher early macro dafa-flow models. 
Neverfheless, we want to point out that the effort required to customize 
an application made by a composition of exisfing skelefon is nof compa¬ 
rable wifh fhe complexify of developing if from scrafch as a sef of macro 
dafa-flow blocks. 

In order fo ease fhe generafion of macro dafa-flow blocks, and fhere- 
fore provide programmers wifh a easier way fo express program-blocks, 
we exploded some metaprogramming techniques fhaf are successfully used 
for code fransformafion in fields like web developmenf and componenf 
based programming dill |79] l97l . Exploifing fhese fechniques fhe pro¬ 
grammers are no longer requesfed fo deal wifh complex applicafion sfruc- 
furing buf simply give hinfs fo fhe mefaprogramming supporf using 
high-level direcfives. The direcfives are used by fhe supporf fo drive 
fhe applicafion fransformafion. Chapfer|^ presenfs our efforfs aimed af 
providing mefaprogramming fools and models for ease fhe generafion 
of macro dafa-flow blocks and fheir run-fime opfimizafion. In parficular, 
fwo resulfs are presenfed. The firsf is "Parallel Absfracfion Layer" (PAL). 
A java armofation ([8) based mefaprogramming framework fhaf resfruc- 
fures applicafions af byfecode-level af run-time in order to make them 
parallel. The parallelization is obtained as 5 mchronously executing the 
armotated methods. Each method call is transformed in a macro dafa- 
flow block fhaf can be dispafched and execufed on fhe available com- 
pufing resources. PAL fransformafions depend on fhe resources avail¬ 
able at run-time, the programmers hints and the available adapters. An 
adapter is a specialized entity that instructs the PAL transformation en¬ 
gine to drive the code transformation depending on the available par¬ 
allel tools and frameworks. The ofher resulf presenfed in fhe chapfer 
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concerns the integration of the Aspect Oriented Programming (l68t l85l 
mechanisms (more in detail the AspectJ framework (|6}) wifh our mod¬ 
ified muskel skeleton framework. The firsf step in fhis direcfion was 
exploifing AspecfJ to implemenf aspect driven program normalization (see 
(HD) in muskel. The second sfep consisfed in testing fhe infegrafion 
of muskel wifh AspecfJ fo in a more complex scenario. Hence, we ex¬ 
ploited fhe aspecf oriented programming supporf infegrafed in muskel 
in order fo develop workflows which sfrucfure and processing are opti¬ 
mized af run-fime depending on fhe available compufafional resources. 
Lef us poinf ouf fhaf we infroduced mefaprogramming techniques for 
easing fhe generafion of macro dafa-flow blocks (in particular fo address 
fhe issue number I) buf as a corollary we obfained fhe possibilify fo opfi- 
mize fhe applicafion and adapf if at run-time with respect to the execut¬ 
ing environment (addressing the issues number III and VI). 

The other two main issues to address are the support for code reuse 
(V) and fhe handling of dynamicify (VII). As we already discussed when 
we infroduced muskel, if addresses fhis lasf poinf fhrough fhe definifion 
of fhe Application Manager. The d 5 mamicify handling is a very imporfanf 
feafure for nexf generafion parallel programming systems, especially for 
fhe ones designed for compufafional Grids. Acfually, muskel frame¬ 
work, af leasf in ifs original form, is designed fo be exploifed in clusfer 
and nefwork of worksfations rafher fhan in Grids. Indeed, some of ifs 
feafures limit its exploitation on Grids, in particular: 

• muskel communicates with the resources it recruits exploiting the 
RMI protocol, that (at least in its original version) uses TGP ports 
that are typically blocked by firewall; 

• fhe compufafional resources are found by muskel exploifing mul- 
ficasf communicafions fhaf are offen blocked by firewall; 

• fhe recruifmenf of a compufafional resource requires fo muskel 
programmers fo run a proper applicafion on fhe resource, hence fo 
have an accounf on if; 

• the Application Manager is a centralized entity. This represents 
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a twofold limitation in Grid environment: it is a single point of 
failure and a bottle-neck that curb the scalability of the approach. 

We addressed most of these limitations exploiting ProActive Parallel Suite 
(11081 to implement the macro data-flow distributed interpreters (see the 
experimental results presented in Chapter |^. ProActive provides mech¬ 
anisms to turmel RMI communications and ease the deployment of Grid 
applications. Indeed, it has been successfully used for developing appli¬ 
cations in the GridSOOO (|2ll platform. ProActive support for Grids has be¬ 
came more complete since it began to support the component based de¬ 
velopment, in particular the support for the CoreGrid Grid Component 
Model (|52|l . Indeed, several studies recognized that component tech¬ 
nology could be leveraged to ease the development of Grid Application 
([25] [72l and a few component based model have been proposed by par¬ 
allel computing scientific community for programming Grids (i5l [52ll67ll . 
Component-based software development can be considered an evolu¬ 
tionary step beyond object-oriented design. Object-oriented techniques 
have been very successful in managing the complexity of modem soft¬ 
ware, but they have not resulted in significant amounts of cross-project 
code reuse. Furthermore, sharing object-oriented code is difficult be¬ 
cause of language incompatibilities, the lack of standardization for inter¬ 
object communication, and the need for compile-time coupling of inter¬ 
faces. Component-based software development addresses issues of lan¬ 
guage independence (seamlessly combining components written in dif¬ 
ferent programming languages) and component frameworks define stan¬ 
dards for communication among components. Finally, the composition 
compatibility is evaluated providing a meta-language specification for 
their interfaces. The GCM represents one of the main European scientific 
community efforts for designing and developing (|3ll a grid component 
model. We contributed to the design of GCM and its reference imple¬ 
mentation together with the research group that developed muskel and 
with several European research groups. In particular, we focused our 
contribution, in the context of the CoreGrid Programming model virtual 
institute, on GCM autonomic features. Therefore, by designing the au¬ 
tonomic features of GCM components, each component is able to react 
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dynamically to changes in the executing environment. We referred to the 
muskel application manager approach, generalizing and extending the 
approach to make it suitable for componenfs based models. Indeed, each 
GCM componenf wifh a complefe supporf of autonomic feafures has an 
Autonomic Manager fhaf observes fhe componenf behavior. In case fhe 
behavior furns ouf to be differenf from fhe one expected fhe manager 
frigger a componenf reconfigurafion. In ofher words, GCM autonomic 
feafures provide programmers wifh a configurable and sfraighfforward 
way to implement autonomic grid applications. Hence, they ease the 
development of applicafion for fhe Grids. Nevertheless, they rely fully 
on fhe applicafion programmer's expertise for fhe sef-up of fhe manage- 
menf code, which can be quite difficulf to wrife since if may involve fhe 
management of black-box componenfs, and, nofably, is failored for fhe 
particular component or assembly of fhem. As a resulf, fhe infroducfion 
of d 5 mamic adapfivify and self-management might enable the manage¬ 
ment of grid d 5 mamism, and uncerfainfy aspecfs buf, at the same time, 
decreases the component reuse potential since it further specializes com¬ 
ponents with application specific managemenf code. In Chapfer we 
propose Behavioural Skeletons as a novel way fo describe aufonomic com¬ 
ponenfs in fhe GCM framework. Behavioural Skelefons aim fo describe 
recurring patterns of componenf assemblies fhaf can be (eifher sfafically 
or d 5 mamically) equipped wifh correcf and effective managemenf sfrafe- 
gies wifh respecf fo a given managemenf goal. Behavioural Skelefons 
help fhe applicafion designer fo i) design componenf assemblies fhaf can 
be effecfively reused, and ii) cope wifh managemenf complexify by pro¬ 
viding a componenf wifh an explicif confexf wifh respecf fo fop-down 
design (i.e. component nesting). We consider the Behavioural Skeletons, 
coupled with the CoreGRID Grid Component, a good structured paral¬ 
lel programming model for handling d 5 mamicify (VII), supporting reuse 
bofh of funcfional and non-funcfional code (V). The model defines char- 
acfers as fhe Skeleton designers and fhe Expert users fhat can design new 
skelefons and cusfomize fhe exisfing ones (II and III), whereas, sfandard 
users can easily (I) exploif fhe exisfing ones. 
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Chapter 3 


Mixing Structured and 
Macro-Dataflow approaches 


Chapter road-map In this chapter we describe our contribution to skele¬ 
ton customization. We start with an introduction on structured programming 
model outlining its main advantages and recalling its main limitations. In par¬ 
ticular, we focus on the skeleton customization issue. Namely the lack of flex¬ 
ibility of skeletal systems in expressing parallel form different from the ones 
"bundled" with the skeleton framework. Then we briefly introduce the data¬ 
flow approach we conceived to address of this limitation and we report related 
work: alternative approaches addressing the structured parallel programming 
limitations (Section \3.1lj .Besides, we introduce classical implementation tem¬ 
plate and more recent data-flow technologies as used to design and implement 
skeleton systems (Section 3.2 1. Then, we describe the details of our contribu¬ 
tion, i.e. our extended version of muskel framework, discussing how skele¬ 
tons customization is supported exploiting data-flow implementation (Section 


3.3.1 L Finally, we report the experimental results we obtained exploiting our 


customized muskel (SectionB^. 
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3.1 Data-flow enables skeleton customization 


We already introduced structured parallel programming models in the 
previous chapter, where we described their Pros and Cons. Let us to 
briefly recall here their main features and limitations. 

Structured parallel programming models provide the programmers 
with native high-level parallelism exploitation patterns that can be in¬ 
stantiated, possibly in a nested way, to implement a wide range of appli- 
cafions (IT3[32ll5ni87ll88]| . In parficular, fhose programming models hide 
fo programmers "assembly level" of parallel programming, i.e. by avoid¬ 
ing a direcf inferacfion wifh fhe disfribufed execufion environmenf via 
communicafion or shared memory access primifives and / or via explicif 
scheduling and code mapping. Rafher, fhe high-level native, paramef- 
ric parallelism exploifafion patterns provided encapsulafe and absfracf 
from all fhese parallelism exploifafion relafed defails. In confrasf, when 
using a fradifional parallel programming sysfem, fhe programmers have 
usually fo explicifly program code for disfribufing and scheduling fhe 
processes on fhe available resources and for moving inpuf and outpuf 
dafa among fhe involved processing elemenfs. The cosf of fhis appealing 
high-level way of dealing wifh parallel programs is paid in ferms of pro¬ 
gramming freedom. The programmer (or skelefon sysfem user) is nor¬ 
mally nof allowed fo use arbitrary parallelism exploitation patterns, but 
he must only use the ones provided by the system. They usually include 
all those reusable patterns that have efficient distributed implementa¬ 
tions available. This is mainly aimed at avoiding the possibly for fhe 
programmers fo write code that can potentially impairs the efficiency of 
the implementation provided for fhe available, native parallel patterns. 
This is a well-known problem (See chapter]^. 

In this Chapter we discuss the methodology we conceived, designed 
and used to modify fhe muskel parallel programming environmenf in 
order fo provide fo programmers fhe possibilify fo mix sfrucfured and 
unsfrucfured ways of expressing parallelism while preserving mosf of 
fhe benefifs fypical of sfrucfured parallel programming models. The 
mefhodology is based on fhe macro dafa-flow model. Sfrucfured parallel 
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exploitation patterns are implemented translating them into macro data¬ 
flow graphs executed by the distributed macro data-flow interpreters. 
Unstructured, user-defined parallelism exploitation patterns are achieved 
by explicitly programming data-flow graphs. These (macro) data-flow 
graphs can be used in the skeleton systems in any place where prede¬ 
fined skeletons can be used, fhus providing fhe possibilify to seamlessly 
infegrafe bofh kind of parallelism exploifafion wifhin fhe same program. 
The mechanisms enabling dafa-flow graphs cusfomizafion provide pro¬ 
grammers fhe possibilify to program new parallelism exploifafion pat- 
fems. 

The mefhodology has been developed fogefher wifh fhe ofher au¬ 
thors of (|2T1 , we all confribufed in a subsfantially equal way to the con¬ 
ception, design and implementation of fhe approach. 

Macro dafa-flow implemenfafion for algorithmical skelefon program¬ 
ming environmenf was infroduced in lafe '90 (l56ll and fhen has been used 
in ofher confexfs related to skelefon programming environmenfs (llOlb . 

Cole eSkel, we already presenfed in fhe previous chapter, addresses 
fhese problems by allowing programmers fo program fheir own peculiar 
MPI code wifhin each process in fhe skelefon free. Programmers can ask 
fo have a sfage of a pipeline or a worker in a farm running on k pro¬ 
cessors. Then, fhe programmer may use fhe k processes communicafor 
refurned by fhe library for fhe sfage/worker fo implemenf ifs own par¬ 
allel pipeline sfage/worker process. As far as we know, fhis is fhe only 
affempf fo infegrafe ad hoc, unsfrucfured parallelism exploifafion in a 
sfrucfured parallel programming environmenf. The implemenfafion of 
eSkel, however, is based on process templates, rafher fhan on dafa flow. 

Ofher skelefon libraries, such as Muesli (I87ll88tl89l , provide program¬ 
mers with a quite large flexibility in skeleton programming following a 
differenf approach. They provide a number of dafa parallel dafa sfruc- 
fures along wifh elemenfary collective dafa parallel operations fhat can 
be arbitrary nested to get more and more complex data parallel skele¬ 
tons. However, this flexibility is restricted to the data parallel part, and 
it is anyway limited by the available collective operations. 
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C02P3S (|92)l is a design pattern based parallel programming envi¬ 
ronment written in Java and targeting symmetric multiprocessors. In 
C02P3S, programmers are allowed to program their own parallel design 
patterns (skeletons) by interacting with the intermediate implementation 
level (l38l . Again, this environment does not use data flow technology but 
implements design patterns using proper process network templates. 

JaSkel (|69]| provides a skeleton library implementing the same skele¬ 
ton set than muskel. In JaSkel, however, skeletons look much more 
implementation templates, according to the terminology used in Section 
|3.2| However, it looks like the programmer can exploit the full OO pro¬ 
gramming methodology to specialize the skeletons to his own needs. As 
the programmer is involved in the management of supporf code foo (e.g. 
he has fo specify fhe masfer process/fhread of a fask farm skelefons) 
JaSkel can be classified as a kind of "low-level, exfensible" skelefon sys¬ 
tem, although it is not clear from fhe paper whefher entirely new skele¬ 
tons can be easily added to the system (actually, it looks like it is not 
possible at all). 

3.2 Template based vs. data-flow based skele¬ 
ton systems 

A skeleton based parallel programming environment provides program¬ 
mers with a set of predefined and paramefric parallelism exploifafion 
patterns. The patterns are paramefric in fhe kind of basic compufafion 
execufed in parallel and, possibly, in the execution parallelism degree or 
in some other execution related parameters. As an example, a pipeline 
skeleton takes as parameters the computations to be computed at the 
pipeline stages. In some skeleton systems these computations can be ei¬ 
ther sequential computations or parallel ones (i.e. other skeletons) while 
in other systems (mainly the ones developed at the very beginning of 
the skeleton related research activity) these computations may only be 
sequential ones. 

The first attempts to implement skeleton programming environments 
all relied on the implementation template technology. As discussed in 
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Optimized pocess network Target architecture 




Figure 9: Skeleton program execution according to the implementation tem¬ 
plate approach. 
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(|94l , in a implementation template based skeleton system each skeletons 
is implemented using a parametric process network picked up among 
the ones available for that particular skeleton and for fhe kind of fargef 
archifecfure af hand in a femplafe library (see (l96l , discussing several 
implemenfafion femplafes, already appeared in bibliography, all suif- 
able fo implemenf fask farms, fhaf is embarrassingly parallel compufa- 
fions implemenfed according fo a masfer-worker paradigm). The fem¬ 
plafe library is designed once and for all by fhe skeleton system designer 
and summarizes his knowledge concerning implemenfation of fhe par¬ 
allelism exploifafion pafferns modeled by skelefons. Therefore, fhe com¬ 
pilation process of a skelefon program, according fo fhe implemenfafion 
femplafe model, can be summarized as follows: 

1. fhe skelefon program is parsed, a skelefon free is derived, hosfing 
fhe precise skelefon sfrucfure of fhe applicafion. The skelefon free 
has nodes marked wifh one of fhe available skelefon, and leaves 
marked wifh sequenfial code (sequenfial skelefons). 

2. fhe skelefon free is traversed, in some order, and templates from fhe 
library are assigned fo each one of fhe skelefon nodes, buf fhe se¬ 
quenfial ones, fhaf always correspond fo fhe execution of a sequen¬ 
fial process on fhe fargef machine. During fhis phase, parameters 
of fhe femplafes (e.g. fhe parallelism degree or fhe kind of com- 
municafion mechanisms used) are fixed, possibly exploiting proper 
heurisfics associated fo fhe library enfries 

3. fhe enriched skelefon free is used fo generafe fhe acfual parallel 
code. Depending on fhe sysfem fhaf may involve a fradifional com¬ 
pilation sfep (e.g. in P3L when using fhe Anaclefo compiler (l47l 
or in ASSIST when using fhe astcc compiler fools (1141 ITSl l or ex¬ 
ploiting proper parallel libraries (e.g. in Muesli (l89t and eSkel (l49l 
exploiting MPI wifhin a proper skelefon library hosfing femplafes 

4. fhe parallel code is evenfually run on fhe fargef archifecfure, possi¬ 
bly exploiting some kind of loader/deploy fool. 
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Figure [^summarizes the process leading from a skeleton source code to 
the rurming code exploiting template technology. 

More recently, an implementation methodology based on data-flow 
has been proposed (l56t . In this case the skeleton source code is used to 
compile a data-flow graph and the data-flow graph is then executed on 
the target architecture exploiting a suitable distributed data-flow inter¬ 
preter engine. The approach has been used both in the implementation 
of Lithium (IT2l 11091 and in Serot's SKIPPER skeleton environment dlOOI I. 
In both cases, the data-flow approach was used to support fixed skeleton 
set programming environments. We adopted the very same implemen¬ 
tation approach to develop our version of the muskel framework, mod¬ 
ifying it in collaboration with the original developers, enriching it with a 
data-flow implementation to support extensible skeleton sets. 

When data-flow technology is exploited to implement skeletons, the 
compilation process of a skeleton program can be summarized as fol¬ 
lows: 

1. the skeleton program is parsed, a data-flow graph is derived. The 
data-flow graph represents the pure data-flow behavior of the skele¬ 
ton tree in the program 

2. for each one of the input tasks, a copy of the data-flow graph is 
instantiated, with the task appearing as an input token to the graph. 
The new graph is delivered to the distributed data-flow interpreter 
"instruction pool" 

3. the distributed macro data-flow interpreter fetches fireable instruc¬ 
tions from the instruction pool and the instructions are executed 
exploiting the nodes in the target architecture. Possibly, optimiza¬ 
tions are taken into account (based on proper heuristics) that try to 
avoid unnecessary communications (e.g. caching tokens that will 
eventually be reused) or to adapt the computation grain of the pro¬ 
gram to the target architecture features (e.g. delivering more than 
a single fireable instruction to remote nodes to decrease the impact 
of communication set up latency, or multiprocessing the remote 
nodes to achieve communication and computation overlap). 
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Pipeline main (...) 
stage1(...) 
stage2(...) 
stage3(...) 
end pipeline 

seq stage1(...) 
{...} 

farm stage2(...) 

seq2(...) 
end farm 



Distributed data fiow interpreter (e.g. Masters-Workers) Target architecture 

Figure 10: Skeleton program execution according to the data-flow approach. 
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Figure [TO] summarizes the process leading from skeleton source code to 
the running code exploiting this data-flow approach. 

The two approaches just outlined appear very different, but they have 
been successfully used to implement different skeleton systems. Let us 
to point out a quite subtle difference in the two approaches. 

On the one side, when using implementation templates, the process 
network eventually run on the target architecture is very close to the one 
the programmer has in mind when instantiating skeletons in the source 
code. In some systems the "optimization" phase of Figure]^ is actually 
empty and the program eventually run on the target architecture is build 
out of plain juxtaposition of the process networks making up the tem¬ 
plates of the skeletons using in the program. Even in case the optimiza¬ 
tion phase do actually modify the process network structure (in Figure 
l^the master/slave service process of the two consecutive farms are op¬ 
timized/ collapsed, for instance), the overall structure of the process net¬ 
work does not change too much. 

On the other side, when a data-flow approach is used the process 
network run on the target architecture is completely different from the 
skeleton tree exposed by programmer in the source code. Rather, the 
skeleton tree is used to implement the parallel computation in a correct 
and efficient way, exploiting a set of techniques and mechanisms that are 
much more close to the techniques and mechanisms used in operating 
systems rather than to those used in the execution of parallel programs, 
both structured and unstructured. Under a slightly different perspective, 
this can be interpreted as follows: 

• skeletons in the program "annotate" sequential code by provid¬ 
ing the meta information required to efficiently implement the pro¬ 
gram in parallel; 

• the support tools of the skeleton programming environment (the 
macro data-flow graph compiler and the distributed macro data¬ 
flow interpreter, in this case) "interpret" the meta information to 
accurately and efficiently implement the skeleton program, exploit¬ 
ing (possibly at run-time, when the target architecture features are 
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known) the whole set of known mechanisms supporting imple¬ 
mentation optimization (e.g. caches, pre-fetching, node multipro¬ 
cessing, etc.). 

Under this perspective, the macro data-flow implemenfafion for parallel 
skeleton programs opens new perspectives in fhe design of parallel pro¬ 
gramming systems where parallelism is deal! wifh as a "non-funcfional" 
feafure, specified by programmers and handled by fhe compiling/run- 
fime supporf fools in fhe more convenienf and efficienf way w.r.f. to 
fhe fargef archifecfure af hand. In fhe following Chapfers of fhis fhesis 
will be presented some fechniques we exploded to provide program¬ 
mers mefhodologies aiming fhe expression of non-funcfional require- 
menfs and fheir run-time enforcement. 


3.3 muskel 

We already introduced muskel and its programming model in the Chap¬ 
ter |2] There we also outlined how we modified muskel, collaboraf- 
ing wifh ifs original developers, in order fo provide programmers wifh 
mechanisms enabling skeleton cusfomizafions. In fhis secfion we give a 
more defailed explanafion bofh of fhe original muskel and of fhe en¬ 
hanced version we proposed. 

muskel is skelefon programming environmenf derived from Lifhium 
m, if provides fhe sfream parallel skeletons of Lifhium, namely sfafe- 
less fask farm and pipeline. These skeletons can be arbifrary nesfed, fo 
program pipelines with farm sfages, as an example, and fhey process a 
single sfream of inpuf fasks fo produce a single sfream of oufpuf fasks. 
muskel implemenfs skeletons exploiting dafa-flow technology and Java 
RMI facilities, muskel programmers can express parallel computations 
simply using the provided Pipeline and Farm classes. For instance, to 
express a parallel computation structured as a two-stage pipeline where 
each stage is a farm, muskel programmers should wrife a code such as 
fhe one of Figure The two classes f and g implemenf fhe Skeleton 
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Skeleton main = 

new Pipeline(new Farm(f), 
new Farm(g)); 

Manager manager = new ManagerQ; 

manager.setProgram(main); 

manager.setContract(new ParDegree(IO)); 

manager.setlnputManager(inputl\/lanager); 

manager.setOutputManager(outputl\/lanager); 

manager.evalO; 


Figure 11: Sample muskel code: sketch of all (but the sequential portions 
of code) fhe coded needed to set up and execute a two-stage pipeline with 
parallel stages (farms). 


interface, i.e. supplying a compute method with the signature 

Object compute(Object t) 

computing f and g respectively. The Skeleton interface represents the 
"sequential" skeleton, that is the skeleton always executed sequentially 
and only aimed at wrapping sequential code in such a way such code 
can be used in other, non-sequential skeletons. 

In order to execute the program, a muskel programmer first sets 
up a Manager object. Then, using proper methods, he specifies the pro¬ 
gram to execute, the performance contract required (in this case, the par¬ 
allelism degree required for the execution), the input data source (the in¬ 
put stream manager, which is basically an iterator providing the classical 
boolean hasNextO and Object nextO methods) and who is in charge of 
processing the output data (the output stream manager, just providing a 
void deliver(Object) method processing a single result of the program). 
Eventually he can ask parallel program execution simply issuing an eval 
call to the manager. When the call terminates, an output file is produced. 


75 




Actually, the eval method execution happens in steps. First, the man¬ 
ager looks for available processing elements using a simplified, mul- 
ficasf based peer-fo-peer discovery profocol, and recruifs fhe required 
remofe processing elemenfs. Each remofe processing elemenf runs a 
dafa-flow inferprefer. Then fhe skelefon program (fhe main of fhe ex¬ 
ample depicfed in Figure [TT| | is compiled info a macro dafa-flow graph 
(acfually capifalizing on normal form resulfs shown in (IT2l [18)) and a 
fhread is forked for each one of fhe remofe processing elemenfs recruifed. 
Then the input stream is read. For each task item, an instance of fhe 
macro dafa-flow graph is creafed and fhe fask ifem foken is stored in 
fhe proper place (inifial dafa-flow insfrucfion(s)). The graph is placed in 
fhe fask pool, fhe reposifory for dafa-flow insfrucfions to be executed. 
Each fhread looks for a fireable insfrucfion in fhe fask pool and deliv¬ 
ers if for execution to fhe associated remofe dafa-flow inferprefer. The 
remofe inferprefer insfance associated fo fhe fhread is initialized by be¬ 
ing sent the serialized code of fhe dafa-flow insfrucfions, once and for 
all before fhe compufation acfually sfarfs. Once fhe remofe inferprefer 
ferminafes fhe execution of fhe dafa-flow insfrucfion, fhe fhread eifher 
stores fhe resulf foken in fhe proper "nexf" dafa-flow insfrucfion(s) in 
fhe fask pool, or if direcfly wrifes fhe resulf fo fhe outpuf sfream, invok¬ 
ing fhe deliver mefhod of fhe oufpuf sfream manager. If a remofe node 
"fails" (e.g. due fo a nefwork failure, or fo fhe node failure/shufdown), 
fhe manager looks for anofher node and sfarfs dispafching dafa flow in¬ 
sfrucfions fo fhe new node insfead (|58) . As fhe manager is a cenfralized 
enfify, if if fails, fhe whole compufation fails. However, fhe manager is 
usually run on fhe machine of fhe muskel user, which is assumed fo be 
safer fhan the remote nodes recruited as remote interpreter instances. 

The policies implemented by the muskel managers are best effort. 
The muskel framework tries to do its best to accomplish user requests. 
In case it is not possible to completely satisfy fhe user requesfs, fhe frame¬ 
work accomplishes fo esfablish fhe closesf configuration to the one im¬ 
plicitly specified by fhe user wifh fhe performance contracf. In fhe exam¬ 
ple above, fhe framework fries fo recruif 10 remofe inferprefers. In case 
only n < 10 remofe inferprefers are found, fhe parallelism degree is sef 
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exactly to n. In the worst case, that is if no remote interpreter is found, fhe 
compufafion is performed sequenfially, on fhe local processing elemenf. 

In fhe currenf version of muskel, fhe only performance confracf ac¬ 
tually implemented is the ParDegree one, asking for fhe usage of a con- 
sfanf number of remofe inferprefers in fhe execufion of fhe program. We 
do nof enfer in more defail in fhe implemenfafion of fhe disfribufed dafa- 
flow inferprefer here. The inferesfed reader can refer fo ( |56ll58l . Insfead, 
we will fry fo give a beffer insighf info fhe compilafion of skeleton code 
info dafa-flow graphs. 

A muskel parallel skelefon code is described by fhe grammar: 

P ::= seq{dassName) \ pipe(P, P) | farm(P) 

where fhe classNames refer fo classes implementing fhe Skeleton inter¬ 
face, and a macro dafa-flow insfrucfion is a fuple: {id, gid, opcode, , O^) 
where id is fhe insfrucfion identifier, gid is fhe graph identifier (bofh are 
eifher integers or fhe special Nold identifier), opcode is fhe name of fhe 
Skeleton class providing the code to compute the instruction (i.e. com¬ 
puting the output tokens out of fhe inpuf ones) and I and O are fhe 
inpuf fokens and fhe outpuf foken desfinafions, respecfively. An inpuf 
token is a pair {value, presenceBit) and an output token destination is 
a pair {destlnstructionld,destTokenNumber). With these assumptions, a 
data-flow insfrucfion such as: 

{a, b, f, ((123, true), (null, false)), {{i,j}}} 


is fhe insfrucfion wifh idenfifier a belonging fo fhe graph wifh idenfi- 
fier b. It has two input tokens, one present (the integer 123) and one not 
present yet. It is not fireable, as one token is missing. When the missing 
token will be delivered to this instruction, coming either from fhe inpuf 
sfream or from anofher insfrucfion, fhe insfrucfion becomes fireable. To 
be computed, fhe two fokens musf be given fo fhe compute method of 
the f class. The method computes a single result that will be delivered to 
the instruction with identifier i in fhe same graph, in fhe position corre¬ 
sponding fo inpuf foken number j. The process compiling fhe skelefon 
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program into the data-flow graph can therefore be more formally de¬ 
scribed as follows. We define a pre-compile funcfion PC\ ] as: 

PC[seq (f)] 9 id = \i.{{newIdQ, gid, f, ((null, false)), {{i, Nold)))} 
PC'[farm(P )],id = C[P]gid 

PCfpipe (Pi, P2)Uci = AiiCiPilgw {getId{C[P2\gid )), C[P2],id{i)} 

where Xx.T is fhe usual funcfion represenfafion {{Xx.T){y) = T\x=y) and 
getID{) is fhe funcfion refuming fhe id of fhe firsf insfrucfion in ifs ar- 
gumenf graph, fhaf is, fhe one assuming fo receive fhe inpuf token from 
oufside fhe graph, and a compile funcfion CQ such as: 

C[P ] = PC[P ]„e»Gid() (Nold) 

where newIdO and newGidO are sfafeful functions refuming a fresh (i.e. 
unused) insfrucfion and graph identifier, respectively. The compile func¬ 
fion refums fherefore a graph, wifh a fresh graph identifier, hosting all 
fhe dafa-flow insfrucfions relative fo fhe skelefon program. The resulf 
fokens are identified as fhose whose desfinafion is Nold. As an example, 
fhe compilation of fhe main program pipe(farm(seq(f)), farm(seq(g))) 
produces fhe dafa flow graph: 

{(1,1, f, ((null, false)), ((2,1))) , (2,1, g, ((null, false)), ((Nold, Nold)))} 

(assuming fhaf idenfifiers and token positions sfarf from 1). 

When fhe applicafion manager is fold fo acfually compufe fhe pro¬ 
gram, via an evalO mefhod call, fhe inpuf file sfream is read looking for 
fasks fo be compufed. Each fask found is used fo replace fhe dafa field of 
fhe lower id dafa-flow insfrucfion in a new C[P ] graph. In fhe example 
above, fhis resulfs in fhe generation of a sef of independenf graphs such 
as: 

{(l,i, f, ((taski,true)), ((2,1))) , (2, i, g, ((null, false)), ((Nold, Nold)))} 

for all fhe fasks ranging from taski fo taskn- 

All fhe resulting insfrucfions are puf in fhe fask pool of fhe disfribufed 
inferprefer in such a way fhaf fhe confrol fhreads faking care of "feeding" 
fhe remofe dafa-flow inferprefer insfances can sfarf fefching fhe fireable 
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Skeleton inci = new lnc(); 

Dest d = new Dest(0, 2 ,Mdfi.NoGraphld); 

Dest[] dests = new Dest[1]; 
dests[0] = d; 

Mdfi i1 = new Mdfi(manager,1 ,inc1,1,1 ,dests); 

Skeleton sql = new Square(); 

Destdl = new Dest(0,Mdfi.Nolnstrld, Mdfi.NoGraphId); 
Dest[] destsi = new Dest[1]; 
dests 1[0] = d1; 

Mdfl i2 = new Mdfl(manager,2,sq1,1,1 ,dests1); 
MdfGraph graph = new MdfGraph(); 
graph.acldlnstructlon(il); 
graph.addlnstructlon(i2); 

ParCompute userDefMDFg = new ParCompute(graph); 


Figure 12: Custom/user-defined skeleton declaration. 


instructions. The output tokens generated by instructions with destina¬ 
tion tag equal to Nold are directly delivered to the output file stream by 
the threads receiving them from fhe remofe inferprefer insfances. Those 
wifh a non-NoId flag are delivered fo fhe proper insfrucfions in fhe fask 
pool fhaf will evenfually become fireable. 

3.3.1 Programmer-defined skeletons 

In order to introduce completely new parallelism exploitation patterns, 
our version of fhe muskel framework provides programmers wifh mech¬ 
anisms fhaf can be used fo design plain, arbitrary macro data-flow graphs. 
A macro dafa-flow graph can be defined creafing some Mdfi (macro 
dafa-flow insfrucfion) objecfs and cormecfing fhem in a MdfGraph ob- 
jecf. As an example, fhe code in Figurej^is fhe code needed fo program 
a dafa-flow graph wifh two instructions. The first one computes the com¬ 
pute method incl on its input token and delivers the result to the second 
instruction. The second one, computes the sql compute method on its 
input token and delivers the result to a generic "next" instruction (this 


79 



is modeled giving the destination token tag a Mdfi.NoInstrld tag). The 
Dest stuff in fhe code is meanf fo represenf desfinafion of oufpuf fokens 
as friples hosfing fhe graph identifier, fhe insfrucfion idenfifier and fhe 
desfinafion inpuf foken fargefed in fhis insfrucfion. Macro dafa-flow in- 
sfrucfions are build sfafing fhe manager fhey refer fo, fheir idenfifier, fhe 
code execufed (musf be a Skeleton object) the number of inpuf and ouf¬ 
puf fokens and a vecfor wifh a desfinafion for each one of fhe oufpuf fo¬ 
kens. Take info accounf fhaf fhe simple macro dafa-flow graph of Figure 


primitive muskel skeleton code such as: 

Skeleton main = new Pipeline(new Inc(), new Sq())) 

More complex, programmer-defined macro dafa-flow graph may com¬ 
prehend insfrucfions delivering fokens fo an arbitrary number of ofher 
insfrucfions, as well as insfrucfions gafhering inpuf fokens from several 
disfincf ofher insfrucfions. 

MdfGraph objecfs are used fo creafe new ParCompute objecfs. The 
ParCompute objecfs can be used in any place were a Skeleton object is 
used. Therefore programmer-defined parallelism exploifafion patterns 
can be used as pipeline sfages or as farm workers, for insfance. The only 
limifafion on fhe graphs fhaf can be used in a ParCompute object consists 
in requiring that the graph has a unique input token and a unique output 
token. 

When executing programs with programmer-defined parallelism ex¬ 
ploifafion patterns fhe process of compiling skeleton code fo macro dafa- 
flow graphs is slightly modified. When an original muskel skeleton is 
compiled, the process described above is applied. When a programmer- 
defined skelefon is compiled, fhe associated macro dafa-flow graph is 
direcfly faken from fhe ParCompute instance variables where the graph 
supplied by the programmer is maintained. Such graph is linked to the 
rest of fhe graph according fo fhe rules relative fo fhe skelefon where fhe 
programmer-defined skelefon appears. To show how fhe whole process 
works, lef us suppose we wanf fo pre-process each inpuf fasks in such a 


12 is acfually fhe very same macro dafa-flow graph derived compiling a 
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Figure 13: Mixed sample MDF graph: the upper part comes from a 
programmer-defined MDF graph (it cannot be derived using primitive 
muskel skeletons) and the lower part is actually coming from a three stage 
pipeline with two sequential stages (the second and the third one) and a 
parallel first stage (the programmer-defined one). 


way that for each task ti a new task 

t'i = hi{fi{U),g2{gi{fiiU)))) 

is produced. This computation cannot be programmed using the stream 
parallel skeletons currently provided by the original muskel. Then we 
want to process the preprocessed tasks through a two-stage pipeline, in 
order to produce the final result. In this case the programmer can set 
up a new graph using a code similar to the one shown in Figure [Tl| and 
then used that new ParCompute object as the first stage of a fwo-sfage 
pipeline whose second sfage happens fo be fhe posfprocessing fwo-sfage 
pipeline. When compiling fhe whole program, the outer pipeline is com¬ 
piled first. As the first stage is a programmer-defined skeleton, ifs macro 
dafa-flow graph is direcfly faken from fhe programmer-supplied one. 
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The second stage is compiled according to the (recursive) procedure pre¬ 
viously described and eventually the (unique) last instruction of the first 
graph is modified in such a way if sends ifs only outpuf token to fhe 
very firsf insfrucfion in fhe second sfage graph. The resulting graph is 
ouflined in Figure [13} 

Making good usage of fhe mechanisms fhaf allow to define new dafa- 
flow graphs, fhe programmer can arrange to express compufations wifh 
arbifrary mixes of arbifrary dafa-flow graphs and graphs coming from 
fhe compilafion of sfrucfured, sfream parallel skelefon compufations. The 
execution of fhe resulting dafa-flow graph is supported by the muskel 
distributed data-flow inferprefer as fhe execufion of any ofher dafa-flow 
graph derived from fhe compilafion of a skelefon program. Therefore, 
fhe customized skelefons are efficienfly executed as fhe skelefons "bun¬ 
dled" wifh muskel. Indeed, in dafa-flow based skelefon systems, as 
we already sfafed when we presented fhem, fhe opfimizafions do nof di- 
recfly depends on fhe skelefon sfrucfure buf on fhe dafa-flow engine ca- 
pabilify of executing fhe macro dafa-flow insfrucfion in an efficienf way. 

In order to allow primitive muskel skelefon usage as code to be exe¬ 
cuted in an insfrucfion of a programmer-defined macro dafa-flow graph 
if is sufficienf fo compile "on fhe fly" fhe primitive skelefon and include 
the result (i.e. the macro data-flow graph) of fhis compilafion in fhe 
programmer-defined macro dafa-flow graph. 

As a final example, consider fhe code of Figure [^ This code acfu- 
ally shows how a new Map2 skelefon, performing in parallel fhe same 
compufation on all the portions of an inpuf vector, can be defined and 
used. If's worfh poinfing ouf how programmer-defined skelefons, once 
properly debugged and fine-funed, can simply be incorporated in fhe 
muskel skelefon framework and used seamlessly, as fhe primitive muskel 
ones, but for fhe facf (as show in fhe code) fhe consfrucfor needs fhe man¬ 
ager as a parameter. This is needed jusf fo be able fo link togefher fhe 
macro dafa-flow graphs generated by fhe compiler and fhose supplied 
by fhe programmer. This feafure has been released by posfponing fhe 
dafa-flow graph creation fo fhe momenf fhe graph needs fo be insfan- 
fiafed affer fhe arrival of a new fask fo compufe, as af fhaf time all fhe 
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public class Map2 extends ParCompute { 

public Map2(Skeleton f, Manager manager) { 
super(null); 

program = new MdfGraph(); // first build the empty graph 
Dest [] dds1 = new Dest[2]; // build the emitter instruction 
dds1[0]=new Dest(0,2); 
dds1[1]=new Dest(0,3); 

Mdfi emitter = new Mdfi(manager, 1, new MapEmitter{2), 1,2, dds1); 
program.addlnstruction(emitter); // add it to the graph 
Dest [] dds2 = new Dest[1]; // build first half map Skeleton node 
dds2[0] = new Dest{0,4); 

Mdfi iff = new Mdfi(manager,2, f, 1, 1, dds2); 
program.addlnstruction(ifl); // add it to the graph 
Dest []dds3 = new Dest[1]; // build second half map Skeleton node 
dds3[0] = new Dest{1,4); 

Mdfi if2 = new Mdfi(manager,3, f, 1, 1, dds3); 
program.addinstruction(if2); // add it to the graph 
Dest[] ddslast = new Dest[1]; 
ddslast[0] = new Dest{0,Mdfi.Nolnstrld); 

Mdfi collector = new Mdfi(manager,4,new MapCollector(), 2, 1, ddslast); 

program.addlnstruction(collector); 

return; 


public class SampieMap { 

public static void main(String[] args) { 

Manager manager = new Manager(); 

Skeleton worker = new FdoubleO; 

Skeleton main = new Map2{worker,manager); 

InputManager inManager = new DoubleVectlM(10,4); 
OutputManager outManager = new DoubleVectOM(); 

ParDegree contract = new ParDegree(IO); 

manager.setlnputManager(inManager); 

manager.setOutputManager(outManager); 

manager.setContract(contract); 

manager.setProgram{main); 

manager.com pute(); 

} 

} 


Figure 14: Introducing a new, programmer-defined skeleton: a map work¬ 
ing on vectors and with a fixed, programmer-defined parallelism degree. 


83 





information necessary to perform graph "conjunction" is available. 

3.4 Experimental results 

To validate our approach we conducted some test with our modified ver¬ 
sion of the muskel framework. The original muskel interpreter engine 
has been left basically unchanged, whereas the part supporting paral¬ 
lelism exploitation pattern programming has been changed to support 
linking of custom MDF graphs to the code produced by the compiler out 
of plain muskel skeleton trees. We used our customized version for 
implementing an application that can not be (at least not easily) imple¬ 
mented using standard (i.e. without our proposed customization sup¬ 
port) skeleton enviromnents. 

Figurej^summarizes the typical performance results of our enhanced 
interpreter. We ran several S 5 mthetic programs using the custom macro 
data-flow graph features introduced in muskel. We designed the pro¬ 
grams in such a way the macro data-flow instructions appearing in the 
graph had a precise "average grain" (i.e. average ration between the time 
spent by the remote interpreter to compute the macro data flow instruc¬ 
tion sent to it, and the time spent in communicating data to the remote 
interpreter plus the time to retrieve the computation results). For each 
test-bed we passed as input parameters to the developed programs IK 
input tasks. 

The results show that when the computational grain is small, muskel 
does not scale well, even using a very small number of remote interpreter 
instances. Indeed, Figure [^clearly shows that when the computational 
grain is 3 the efficiency rapidly decreases, going under 0.7 even when 
only four computational resources are used. When the grain is 70 the 
efficiency goes under 0.8 only when the number of recruited computa¬ 
tional resources is higher than 14. Finally, when the grain is high enough 
(about 200 times the time spent in communications actually spent in com¬ 
putation of MDF instructions) the efficiency is definitely close to the ideal 
one even using 16 or more machines. 

Despite the data shown refers to some S 5 mthetic computations, actual 
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Figure 15: Effect of middleware: scalability of the muskel prototype using 
plain RMI vs. the one using ProActive active objects 


computations (e.g. image processing ones) achieved very similar results. 
This because the automatic load balancing mechanism implemented in 
the muskel distributed interpreter, obtained by mean of auto schedul¬ 
ing techniques, perfectly optimized the execution of variable grain MDF 
instructions. All the experiments have been performed on a Linux (ker¬ 
nel 2.4.22) RLX Pentium III blade architecture, with Fast Ethernet inter- 
cormection among the blades, equipped with Java 1.4.1_01 run-time. 

Despite measuring scalability of our modified muskel framework, 
we also have taken into account the possibility to use different mecha¬ 
nisms to support distributed data-flow interpreter execution. In partic¬ 
ular, we investigated the possibility of implementing the muskel ap¬ 
proach for skeleton customization on top of the ProActive framework 
( 11081 both to be able to target a different set of architectures and to demon¬ 
strate the "portability" of our approach, i.e. that it is a feasible and ef¬ 
ficient solution not only when it exploits the muskel data-flow inter¬ 
preter. 
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For this purpose, we conducted some experiments aimed at verifying 
the overhead introduced by Pro Active with respect to the plain Java RMI 
muskel protot 5 rpe, when using the secure shell (ssh) tunneling of fhe 
RMI protocol (feafure nafively provided by fhe ProAcfive framework). 
In particular, we modified fhe "kernel" of fhe dafa-flow inferprefer of 
muskel in order fo make if able to exploif fhe ProAcfive acfive objecfs in 
place of plain RMI objecfs as remote dafa-flow inferprefer insfances. The 
resulfs we achieved are summarized in Figure The figure plofs fhe 
completion times for fhe very same program run on a Linux worksfafion 
clusfer when using plain Java RMI and when using ProAcfive acfive ob¬ 
jecfs fo implemenf fhe remote dafa-flow inferprefer insfances. The macro 
dafa-flow insfrucfions, in fhis case, have a grain comparable fo fhe "high 
grain" of insfrucfions of Figure Experimenfs showed fhaf ProAcfive 
acfive objecfs are slighfly less efficienf buf fhe difference is negligible. In 
fhis case, fhe sefup fime of fhe remofe dafa-flow inferprefer insfances was 
nof considered in fhe overall completion fime, being paid once and forall 
when fhe system is sfarfed up. 
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Efficiency Compietion time (secs) 



Remote interpreter number 



Remote interpreter number 


Figure 16: Scalability of the muskel prototype and effect of computation 
grain. 
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Summarizing the Chapter 


In this Chapter we discussed a methodology for extending algorithmic skeletons 
based parallel programming frameworks aimed at providing programmers with 
the possibility to freely customize the structure of their parallel applications. It 
is based on mechanisms allowing programmers to modify the data-flow graph 
derived from the compilation of skeleton based application. In particular, we dis¬ 
cussed how we modified the muskel framework for parallel programming. The 
version we developed (collaborating with the team that developed the original 
muskel) supports extendability of the skeleton set, as advocated by Cole in his 
"manifesto" paper (l57T >. In particular, we discussed how our modified muskel 
supports the introduction of new skeletons, modeling parallelism exploitation 
patterns not originally covered by the primitive muskel skeletons. This possi¬ 
bility is supported by allowing programmers to define new skeletons providing 
the arbitrary data-flow graph executed in the skeleton and by letting muskel 
to seamlessly integrate such new skeletons in the primitive ones. We also pre¬ 
sented experimental results validating our muskel approach to extend and 
customize its skeleton set. As far as we know, this is the most significant effort 
in the skeleton community to tackle problems deriving from a fixed skeleton set. 
Only Schaeffer and his group at the University of Alberta implemented a system 
were programmers can, in controlled ways, insert new parallelism exploitation 
patterns in the system (l3^ , although the approach followed here is a bit differ¬ 
ent, in that programmers are encouraged to intervene directly in the run-time 
support implementation, to introduce new skeletons, while in our muskel new 
skeletons may be introduced using the intermediate macro data-flow language 
as the skeleton "assembly" language. 
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Chapter 4 

Metaprogramming 
Run-time Optimizations 


Chapter road-map This Chapter presents our ejforts aimed at exploiting 
metaprogramming techniques for optimizing at run-time the execution of struc¬ 
tured parallel applications. The approaches are based on the run-time genera¬ 
tion of macro data-flow blocks from the application code. We start presenting 
the motivations (Section 4.1 ' of our contributions. Then we present PAL (Sec¬ 
tion 4.2( our first result in the field. PAL is a metaprogramming engine that 
transforms at run-time an annotated sequential java code in a parallel program, 
exploiting both programmer hints and executing platform information. We de¬ 
scribe our PAL prototype implementation (Section 4.2.1 1 and the results of the 
tests we made with it (Section 4.2.2 L After we discuss the motivations that 
convinced us to integrate the PAL approach with our version of the muskel 
framework (Section 4.2.3\ . In the following section t\4.3\ we describe the prelim¬ 
inary attempts we made integrating metaprogramming techniques in muskel. 
In Section 4.4 we present how we further enhanced muskel making it able to 
exploit metaprogramming for run-time code optimizations. In particular, how it 
can be exploited to optimize the parallel execution of computations expressed as 
workflows. In Section 4.4.2 we describe the implementation of workflows trans¬ 
formations and in Section 4.4.3 we present the performance results obtained. 
Finally, we compare the two approaches (Section 4.5* and we summarize the 
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Chapter contributions. 


4.1 Our efforts in run-time optimization 

In the previous chapter we described how the macro data-flow model 
can be exploited in order to allow the customization of algorifhmic skele- 
fons. We showed how we modified fhe muskel parallel framework in 
order fo provide programmers wifh mechanisms able fo change skele- 
fons sfrucfure. In fhis chapfer we presenf fhe mefaprogramming fech- 
niques we exploifed bofh fo ease fhe generafion of fhe macro dafa-flow 
graph and fo opfimize af run-fime fhe parallel execution of fhe macro 
dafa-flow blocks. 

4.1.1 Metaprogramming 

Code-generating programs are sometimes called mefaprograms; writing 
such programs is called mefaprogramming. Mefaprograms do parf of fhe 
work during compile-fime fhaf is ofherwise done af run-fime. Compile- 
fime mefaprogramming exploifs informafion available af compile-fime 
fo generafe femporary source code, which is merged by fhe compiler 
wifh fhe resf of fhe source code and fhen compiled. The goal of run-fime 
mefaprogramming, insfead, is fo achieve real-fime code opfimizafions 
fransforming or adapting fhe code whenever some informafion becomes 
available. 

Compile-time metaprogramming 

The most common metaprogramming tool is a compiler, which allows 
a programmer to write a relatively short program in a high-level lan¬ 
guage and uses it to write an equivalent assembly language or machine 
language program. Another still fairly common example of mefapro¬ 
gramming mighf be found in fhe use of Templafe Mefaprogramming. 
Templafe mefaprogramming is a mefaprogramming fechnique in which 
femplafes are used by a compiler fo generafe femporary source code, 
which is merged by fhe compiler wifh fhe resf of fhe source code and 
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then compiled. The output of these templates includes compile-time con¬ 
stants, data structures, and complete functions. The use of femplafes can 
be fhoughf of as compile-fime execufion. The fechnique is used by a 
number of languages, fhe mosf well-known being C++, buf also D, Eif¬ 
fel, Haskell, ML and XL. The use of femplafes as a mefaprogramming 
fechnique requires two disfincf operafions: a femplafe musf be defined, 
and a defined femplafe musf be insfanfiafed. The femplafe definifion 
describes fhe generic form of fhe generafed source code, and fhe in- 
sfanfiafion causes a specific sef of source code fo be generafed from fhe 
generic form in fhe femplafe. Templafe mefaprogramming is generally 
Turing-complefe, meaning fhaf any compufafion expressible by a com- 
pufer program can be compufed, in some form, by a femplafe mefapro- 
gram. Templafes are differenf from macros. A macro, which is also a 
compile-fime language feafure, generafes code in-line using fexf manip¬ 
ulation and subsfifufion. Macro sysfems offen have limifed compile-fime 
process flow abilities and usually lack awareness of fhe semantics and 
fype sysfem of fheir companion language (an exception should be made 
wifh Lisp's macros, which are written in Lisp ifself, and is nof a sim¬ 
ple fexf manipulation and subsfifufion). Templafe mefaprograms have 
no mufable variables fhaf is, no variable can change value once if has 
been initialized, fherefore femplafe mefaprogramming can be seen as a 
form of functional programming. In facf, many femplafe implemenfa- 
fions only implemenf flow confrol fhrough recursion. Some common 
reasons fo use femplafes is fo implemenf generic programming (avoid¬ 
ing secfions of code which are similar excepf for some minor variafions) 
and especially fo perform aufomafic compile-fime opfimizafion such as 
doing somefhing once af compile-fime rafher fhan every time fhe pro¬ 
gram is run, for insfance having fhe compiler unroll loops fo eliminafe 
jumps and loop counf decremenfs whenever fhe program is execufed. 
The main problem of fhis approach is fhe inefficienf exploifafion of fhe 
executing environmenf. Indeed fo guaranfee fhe code porfabilify such 
optimizations are done in a generic way, for insfance wifhouf exploifing 
specific CPU exfension like SSE or 3DNow. To overwork if fhe applica- 
fion should be re-compiled once all fhe rurming archifecfure defails are 
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known. 


Run-time metaprogramming 

Run-time metaprogramming points at either the generation of programs 
specialized with respect to the running architecture or the adaptation of 
programs wifh respecf fo addifional informafion provided by program¬ 
mers, e.g. non-funcfional requiremenfs. The mefaprogramming relafed 
informafion (mefadafa) is processed by fhe mefaprogramming run-fime 
support. It exploits both such metadata and the environmental informa¬ 
tion to transforms fhe original code info an opfimized one. Neverfheless, 
fhis solufion presenfs a major problem: fhe re-compilafion overhead. In¬ 
deed, re-compile fhe whole application from scrafch on each machine if 
is moved for execufion is compufationally expansive. A viable solufion 
consisfs in writing the applications using bytecode based languages, like 
Java and .NET. Indeed, their compilers do not translate the program into 
target machine language but translate it into an intermediate language 
(IL). The IL has greater expressiveness than the machine and the assem¬ 
bly languages and can be transformed in a machine-level program pay¬ 
ing a small overhead. Furfhermore, fhere are ofher advanfages in imple- 
menfing applicafion, especially fhe disfribufed ones, exploifing a virfual 
machine based language: e.g. fhe possibilify fo run programs across dif- 
ferenf plafforms af fhe only cosf of porfing fhe execufion environmenf 
and fo achieve beffer securify (fhe execufion engine mediafes all accesses 
fo resources made by programs verifying fhaf fhe sysfem can nof be com¬ 
promised by fhe running applicafion). 

In fhe pasf, ofher programming languages wifh fhe same archifecfure, 
essenfially p-code, have been proposed (see for insfance fhe infroducfion 
of ([86J l buf Java has been fhe firsf fo have a huge impacf on program¬ 
ming mainsfream. Java approach has been recognized as successful, 
indeed, since fhe 2002 also Microsoff infroduced fheir virfual-machine 
based programming languages. They are based on fhe Common Lan¬ 
guage Infrasfrucfure (CLI). The core of CLI is fhe virfual execufion sys¬ 
fem also known as Common Language Runfime(CLR). Bofh JVM (l9Tt 
and CLR (0 implemenf a mulfi-fhreaded sfack-based virfual machine. 
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that offers many services such as d 5 mamic loading, garbage collecfion, 
clearly fhe Jusf In Time (JIT) compilafion and above all a nofeworfhy re¬ 
flection supporf. Feafures like garbage collecfion raise fhe programming 
absfracfion level whereas dynamic loading, JIT compilafion and a nafive 
mulfi-fhread supporf simplify fhe fask of programming disfribufed and 
concurrenf applications. Retiecfion supporf enables programs fo read ifs 
own mefadafa. A program retiecfing on ifself exfracf mefadafa (from ifs 
represenfafion expressed in ferms of infermediafe language) and using 
thaf mefadafa can modify ifs own behavior. Retiecfion support is useful 
fo inspecf fhe sfrucfure of f 5 rpes, fo access fields and even fo choose dy¬ 
namically fhe mefhods fo invoke. Exploiting retiecfion supporf programs 
can change fheir sfrucfure and their (byte) code. The reflection support 
can be provided by the run-time system at different levels of complexify 
(136): 

• Infrospecfion : fhe program can access fo a represenfafion of ifs 
own infernal sfafe. This supporf may range from knowing fhe f5T’e 
of values at run-time to having access to a representation of fhe 
whole source program. 

• Intercession : the representation of fhe sfafe of fhe program can be 
changed af run-time. This may include the set of fypes used, values 
and fhe source code. 

Bofh infrospecfion and infercession require a mechanism, called reifica¬ 
tion, fo expose fhe execufion sfafe of a program as dafa. The reificafion 
mechanism exposes an absfracfion of some elemenfs of fhe execufion en- 
vironmenf. These elemenfs may include programming absfracfions such 
as fypes or source code; fhey may also include ofher elemenfs, like fhe 
evaluation stack (as in 3-LISP dlOSH , that are not modeled by the lan¬ 
guage. For compiled languages it could be harder to reflect elements of 
fhe source language: fhe objecf program runs on a machine fhaf usually 
is far from fhe absfracf machine of fhe source language. Enabling RTTI 
(Runtime Type Identification, a support that allows a program to have 
exact information about type of objecfs af run-fime) in C++, for insfance, 
requires fhaf fhe run-fime supporf confain additional code fo keep hack 
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of types at run-time. Besides, the programmer would expect abstractions 
compatible with the structure of the programming language abstract ma¬ 
chine (unless he is interested in manipulating the state of the machine 
that is target of the compilation). 

Custom metadata management 

The metadata readable through the advanced reflection supports are both 
the information about t 5 p)es (class, method, field names an hierarchies) 
and about additional, non-functional attributes. A straightforward ex¬ 
ample is the Java serialization architecture: the programmer can declare 
the instances of a serializable class simply by implementing the Serializ¬ 
able interface, which in fact is an empty interface. Thus, two types that 
differ only for the implementation of the Serializable interface are in¬ 
distinguishable from the execution (functional) standpoint. Besides, the 
serialization of the instances of non-serializable t 5 p>es will not be allowed 
by the serialization support. Clearly, this "interface-based" mechanism 
for the metadata specification is not flexible and can not be expressed at 
more fine level, for instance at method-level. This limitation leads to the 
development of Java armotations (|8). A Java armotation is a special S 5 m- 
tax that adds metadata to Java source code. Armotations can be added 
to program elements such as classes, methods, fields, parameters, local 
variables, and packages. Unlike Javadoc tags, Java armotations are reflec¬ 
tive in that they may be retained by the Java VM and made retrievable at 
run-time. The possibility to retain and retrieve this information at run¬ 
time makes the "real" difference between the Java armotations and the 
earlier armotation based approach. For instance, the OpenMP pragma 
based approach or the HPF armotation or consisting in simple directives 
to compiler driving the data decomposition optimization, approaches 
that are not designed to work with non-shared memory architectures. 

The exploitation of Java armotations as a way to embed non-functional 
information is at the base of Attribute Oriented Programming (I98llll4ll . 
Attribute Oriented Programmers use Java annotations to mark program 
elements (e.g. classes and methods) to indicate that they maintain the 
application-specific or domain-specific semantics. As an example, some 
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programmers may define a "logging" attribute and associate it with a 
method to indicate the method should implement a logging function, 
while other programmers may define a "web service" attribute and as¬ 
sociate it with a class to indicate the class should be implemented as a 
web service. Attributes aim the separation of concerns: application's 
core logic (or business logic) are clearly distinguished from application- 
specific or domain-specific semantics (e.g. logging and web service func¬ 
tions). By hiding the implementation details of those semantics from 
program code, attributes increase the level of programming abstraction 
and reduce programming complexity. The program elements associated 
with attributes are transformed in order to fit the programmers' require¬ 
ments. 


The effectiveness of the approach is demonstrated by its rapidly dif¬ 
fusion, indeed some very popular and widely used programming frame¬ 
works dTOl [79l adopted the Attribute Oriented Programming approach 
as a way to embed programmers' hints and requirements. There are also 
some scientific works exploiting annotations information to drive the ap¬ 
plication run-time transformation, for instance in (|45]| authors propose a 
way to transform an annotated application in a multithreaded one and 
&7\ describes a way to transform a POJO in a Fractal component simply 
transforming the code according to the programmer annotations. 

In Section 4.2 we describe how we exploited the Attribute Oriented 
Programming approach in our Parallel Abstraction Layer (PAL). PAL is a 
metaprogramming engine able to d 5 mamically restructure parallel appli¬ 
cations depending both on the information gathered at run-time about 
the rurming platform and on the hints specified inside the source code 
by programmers. 


A slightly different approach that aims to a clear separation between 
the application business code and application management information 
is the Aspect Oriented Programming (AOP) model. Whereas the At¬ 
tribute Oriented Programming model separates the management code 
from the business one exploiting a language support. Aspect Oriented 
Programming model requires programmers provide additional files con¬ 
taining a set of rules which describe the actions to perform when the 
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application execution flow reach certain points. The main actions per¬ 
formed consist in code injection and code substitution. Some scientific 
works exploif AOP for code fransformations. Sobral ef al. discussed fhe 
usage of AOP fo supporf modular computing (l53l 1106111071 . They use 
AOP fechniques fo separafely solve partition, concurrency and disfribu- 
fion problems and evenfually show how fhe relafed aspecfs can be used 
fo provide a (kernel for a) general purpose, modular parallel computing 
framework. Ofher aufhors (0^ demonsfrafed fhaf AOP can be efficienfly 
exploifed in conjuncfion wifh componenfs and patterns fo derive parallel 
applicafions for disfribufed memory sysfems. If highly relies on fhe abil- 
ify of fhe programmer fo find ouf fhe righf places fo exploif aspecfs. In 
((78) anofher approach exploiting aspecfs fo parallelize Java applicafions 
from fhe Java Grande forum using AspecfJ is presenfed. Good resulfs are 
shown in fhe paper, buf fhe procedure used fo exploif aspecfs requires 
enfering fhe program defails fo find ouf possibilifies for parallelizafion. 

In fhe Secfions 4.3 and 4.4 we describe how we infegrafed fhe AOP 
approach in our nexf generation muskel. In particular, how we ex¬ 
ploifed fhe AspecfJ (|6) fool fo manage fhe generation of macro dafa-flow 
blocks, aimed af fhe parallelizafion of workflow compufafions. 

Bofh fhe PAL and fhe AspecfJ infegrafion wifh muskel approaches 
have been published, respectively in (161) and (l60) . In bofh fhe cases fhe 
aufhors collecfively confribufed fo fhe paper. 


4.2 The PAL experience 

The Parallel Absfracfion Layer is a general-purpose approach for imple¬ 
menting simple parallel applicafions fhaf does nof require complex ap¬ 
plication sfrucfuring by programmers. Programmers are only required 
fo inserf, in fhe source code, some hinfs, evenfually exploifed by fhe 
PAL run-fime supporf fo fransform fhe applicafion code. The fransfor- 
mafion is aimed af in enforcing an efflcienf parallel (even disfribufed) 
execution of fhe applicafion. The general idea is ouflined in Figure [T7[ 
Programmers' hinfs consisf in non-funcfional requiremenfs, namely, re- 
quiremenfs which specify criferia fhaf can be used fo judge fhe operafion 
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Figure 17: PAL approach overview 


of a system, rather than specific behaviors. Examples of non-functional 
requirements includes: Efficiency, Price, Hardware Reliability, Software 
and tools availability and Parallelism degree. In PAL implementation 
they are specified through the annotation mechanisms provided by Java 
®. The PAL run-time support exploits the information conveyed in the 
armotations to transform the original program in a parallel one. The 
transformed program is optimized with respect to the target parallel / distributed 
architecture. 


Programmers are required to give some kind of "parallel structure" to 
the code directly at the source code level, as it happens in the algorithmic 
skeleton case. In our PAL implementation it can be done exploiting the 
java annotation mechanism. Por instance, the farm semantics is obtained 
indicating which "parts" of code should be replicated and executed in 
parallel. A "part" is intended to be a piece of side-effect free code which 
input and output data are well-defined. Programmers are in charge of 
ensuring the "parts" satisfy these requirements. Each java code "part" 
is transformed by the PAL in a macro data-flow block that can be dis¬ 
patched for execution. 

PAL has a multi-level software architecture. It is depicted in Pigure 


19 On top, there is PAL frontend, namely the annotations provided by 


PAL and the host language, Java in our PAL implementation. In the bot- 
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public class Mandelbrot{ 

public void paint(GraphicsContext gcont) { 

// computing image size 

Vector<PFFuture<Vector<Vector<lnteger»» man = 

new Vector<PFFuture<Vector<Vector<lnteger»»(numOfLines): 

for{int i=0:i<numOfLines:i++) 
man.add(createLines(...); 

} 

@Parallel(parDegree=16) 

public PFFuture<Vector<Vector<lnteger»> createLines (params ...){ 

Vector<Vector<lnteget» V = new Vector<Vector<lnteger»(): 

// compute points ... 
for (int i = 0; i<cls; i++) { 

v.add(point): 

} 

return new PFFuture<Vector<Vector<lnteger»>(v): 

} 

} 

public class Main { 

public static void main(String[] args) { 

Class [] toBeTransformed = new Class[2]: 
toBeTransformed[0] = Main.class; 
toBeTransformed[1] = Mandelbrot.class; 

PAL.transform(toBeTransformed,args): 

Mandelbrot mBrot = new Mandelbrot(); 

Bufferedimage bi = new Bufferedlmage(2400,1600,TYPE_INT_BGR); 

mBrot. paint{GraphicsEnvironment.getLocalGraphicsEnvironment().createGraphics(bi)); 

} 

} 


Figure 18: Sample code using PAL 


tom layer, there are the adapters and the information system: the formers 
fosfer PAL during code fransformafion insfrucfing if abouf how fo sfruc- 
fure fhe applicafion code fo make if parallel and complianf wifh a specific 
parallel framework. The laffer is a sef of fools aimed af run-fime informa- 
fion gafhering. Finally, fhe middle layer is fhe real mefaprogramming en¬ 
gine fhaf uses fhe informafion gafhered in order fo decide which adapfer 
exploif among fhe available fo enforce fhe non-funcfional requiremenfs 
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expressed by the programmers through annotations. 



Figure 19: PAL architecture 

Compared with traditional skeletal environments, PAL presents three 
additional advantages. 

• First, annotations can be ignored and the semantics of the original 
sequential code is preserved. This means that the programmers' 
application code can be run through a classical sequential compiler 
(or interpreter) suite and debugged using normal debugging tools. 

• Second, annotations are processed at run-time, t 5 rpically exploiting 
reflection properties of fhe hosting language. As a consequence, 
while handling armofafions, a bunch of knowledge can be exploded 
which is nof available af compile-fime (kind of machines af hand, 
kind of infercormecfion nefwork, efc.) and fhis can lead fo more 
efficienf parallel implemenfafions of fhe user application. 

• Third, fhe knowledge concerning fhe kind of fargef archifecfure 
can be exploifed leading fo radically diverse implemenfafion of fhe 
very same user code. As an example, if fhe run-fime can figure 
ouf fhaf fhe fargef archifecfure where fhe program is rurming hap¬ 
pens fo be a grid, if can fransform fhe code in such a way possibly 
coarser grain parallelism is exploifed. On fhe ofher hand, in case 
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the run-time figures out that user asked to execute the code on a 
SMP target, a more efficient, possibly finer grain, mulfifhreaded 
version of fhe code can be produced as fhe resulf of fhe annofafion 
handling. 

PAL enforces code optimizations via aufomafic application resfruc- 
furing in order fo exploif all fhe available application parallelism wifh re- 
specf fo programmer's armofafions (non-funcfional application require- 
menfs). The fransformafion process is done af run-time, which is at the 
time we have the information we need to optimize the restructuring pro¬ 
cess with respect to the available parallel tools and underlying resources. 
The code is transformed af byfecode level fhus, if does nof need fo re¬ 
compile fhe applicafion source code on fhe fargef archifecfure. Hence, 
the transformation introduces only a small overhead for fhe code frans- 
formafions. 

The generative (l54t mefaprogramming engine of PAL gafhers af run- 
fime information on available parallel fools and compufafional resources. 
Then, if analyzes fhe byfecode looking for programmer armofafions (non- 
funcfional requiremenfs) and fransforms fhe annofafed original code fo 
an opfimized, parallel one. The sfrucfure of fhe fransformed byfecode 
depends on fhe selecfed parallel framework (clearly subjecfed fo adapfers 
availabilify) and on fhe presence and / or value of some non-funcfional 
requiremenfs. 

PAL exploifs fhe available parallelism by as 5 mchronously execufing 
parfs of fhe original code. The parfs fo be execufed asynchronously are 
individuafed by fhe annofafions specified by programmers. In particular, 
in Java fhe mosf nafural choice consisfs in individuating mefhods calls 
as fhe parfs fo be as 5 mchronously execufed. As 5 mchronous execufion of 
mefhod code is based on fhe concepf of future (I43ll44ll . When a mefhod 
is called as 5 mchronously if immediafely refums a fufure, fhaf is a sfub 
"empfy" objecf. The caller can fhen continue ifs own compulations and 
access fo fhe fufure objecf confent (e.g. calling ifs mefhods) jusf when 
needed. If in fhe meanwhile fhe refum value has already been compufed, 
fhe call fo reify fhe fufure succeeds immediafely, ofherwise if blocks until 
fhe acfual refurn value is compufed and fhen refurns if. 
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In our PAL implementation, to indicate a method as "parallelizable" 
PAL programmers have simply to put a proper ©Parallel annotation 
enriched with non-functional requirements, such as the required par¬ 
allelism degree, on the line right before mefhod declarafion. Exploif- 
ing fhe armofafion mechanism allows fo keep fhe PAL applicafions very 
similar fo normal sequenfial applicafions, acfually. Hence, Programmers 
may simply run fhe application fhrough sfandard Java fools fo verify if 
is functionally correcf. PAL aufonomically performs af run-fime acfivi- 
fies aimed af achieving fhe as 5 mchronous and parallel execution of fhe 
PAL-annofafed mefhods and af managing any consisfency relafed prob¬ 
lems, wifhouf any furfher programmer infervenfion. The PAL approach 
also avoids fhe proliferation of source files and classes, fhaf is a quife 
common sifuafion in framework based programming, as if works frans- 
forming byfecode. Unforfunafely, if raises several problems relafed fo 
dafa sharing managemenf. As an example, mefhods annofafed wifh a 
©Parallel should nof access class fields: fhey may only access fheir own 
paramefers and fhe local mefhod variables. This is due fo fhe impossibil- 
ify fo infercepf all fhe accesses fo non-privafe class fields. This limifafion 
prevenf fhe usage of sfafic class fields as a way for sharing dafa among 
differenf insfances of annofafed mefhod calls, making more complex fhe 
developmenf of application in which fhe compufafional resources run¬ 
ning fhe differenf annofafed mefhod calls need fo exchange dafa during 
fhe mefhod compufafion. If is worfh fo nofe fhaf fhis is nof a limifafion of 
fhe approach buf depends by fhe Java language. Indeed having a proper 
language supporf for defecfing public field changes if would nof be dif- 
ficulf fo provide a proper armofafion for managing fhe remofe accesses 
fo fields. 

4.2.1 PAL: implementation details 

We implemented a PAL prototype in Java 1.5, as Java provides a man¬ 
ageable intermediate language 0ava bytecode jllOt ) and natively sup¬ 
ports code annotations, since version 1.5. Furthermore, it owns all the 
properties needed by our approach (e.g. t 5 q)e safety and security). For 
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this implementation we developed two distinct adapters. One for trans¬ 
forming fhe byfecode in a mulfifhreaded one and anofher fo fransform 
fhe byfecode making if complianf wifh JJPF. In order fo do fhis our PAL 
implemenfafion makes beffer usage of ASM dlOt : a Java byfecode manip¬ 
ulation framework. 

The currenf PAL profof 5 rpe accepfs only one kind of non-funcfional 
affribufe fhat can be specified wifh fhe ©Parallel armofation: parDe- 
gree. If denofes fhe number of processing elemenfs fo be used for fhe 
mefhod execution. PAL uses such informafion fo make a choice between 
the multithreaded and JJPF adapter. This choice is driven by the number 
of processors/cores available on fhe hosf machine: if fhe machine owns 
a sufficienf number of processors fhe armofafed byfecode direcfly com¬ 
piled from user code is fransformed in a semantically equivalent mul¬ 
tithreaded version. Otherwise, PAL chooses to transform fhe compiled 
byfecode in a semantically equivalent JJPF version that uses several net¬ 
worked machines to execute the program. PAL basically transforms code 
in such a way fhe armofafed mefhods can be compufed as 5 mchronously. 
The original code is "adapfed" using an adapfer in order fo be com¬ 
plianf wifh fhe parallel framework associafed wifh fhe adapfer. In our 
implemenfafion, where fhe only available adapfer for disfribufed com- 
pufafions is fhe JJPF one, fhe mefhods are adapfed fo be run on fhe re- 
mofe JJPF servers displaced onfo fhe processing elemenfs. Conversely, 
fhe main code invoking fhe ©Parallel mefhods is used fo implemenf fhe 
"clienf" code, i.e. fhe applicafion fhe user runs on ifs own local machine. 
This applicafion eventually will interact with the remote JJPF servers 
according to proper JJPF mechanisms and protocols. Method call pa¬ 
rameters, the input data for fhe code fo be execufed as 5 mchronously, are 
packaged in a "fask". When a server receives a fask to be computed, it 
removes its server-descriptor from fhe processing elemenfs available for 
JJPF. When fhe fask compufafion is complefed fhe server re-inserfs ifs 
descriptor from the available ones. In other words, when a armotated 
method is called an empty future is immediately returned, a "task" is 
generated and it is inserted into the JJPF queue; eventually it is sent to 
one among the available processing element, which remove itself from 
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the available resources, computes the task and returns the result that JJPF 
finally put inside the proper future. This implementation schema looks 
like very close to a classical master/slave implementation. 

We could have developed an adapter for ofher parallel programming 
frameworks as fargefs. As an example, we could have used fhe Globus 
foolkif. However, JJPF is very compacf and required a slighfly more com- 
pacf amounf of code fo be fargefed, wifh respecf fo fhe Globus or ofher 
grid middleware frameworks. As fhe principles driving fhe generafion 
of fhe parallel code are fhe same bofh using JJPF and ofher grid middle¬ 
ware frameworks, we preferred JJPF fo be able fo implemenf a proof-of- 
concepf adapfer profof 5 rpe in a very shorf fime. 

As we already sfafed before, our currenf PAL profof 5 q)e has some 
limifafions, in parficular, fhe only paramefer passing semanfics available 
for annofafed mefhods is fhe deep-copy one, and fhe program sequenfial 
semanfics is nof guaranfeed if fhe class fields are accessed from inside 
fhe PAL-annofafed mefhods. 

Figure]^ shows an example of PAL profofype usage, namely a pro¬ 
gram compufing fhe Mandelbrof sef. The Mandelbrot class uses a ©Par¬ 
allel annotation to state that all the createLines calls should be com¬ 
puted in parallel, with a parallelism degree equal to 16. Observe that, 
due to some Java limitations (see below), the programmer must specify 
PFFuture as refum f 5 q?e, and consequenfly refurn an objecf of fhis t5q3e. 
PFFuture is a femplafe defined by fhe PAL framework. If represenfs a 
confainer needed fo enable fhe fufure mechanism. The t 5 q)e specified as 
argumenf is fhe original mefhod refum fype. Inifially we fried fo have 
fo a more fransparenf mechanism for fhe fufure implemenfafion, wifhouf 
any explicif Fufure declarafion. If consisfed in fhe run-fime subsfifufion 
of fhe refum f5q5e wifh a PAL-fype inherifing from fhe original one. In 
our idea, fhe PAL-f 5 rpe would have filfered any original fype derefer- 
enfiafion following fhe wait-by-necessity (|42J semanfics. Unforfunafely, 
we had fo face two Java limifafions fhaf limif fhe currenf profof 5 rpe fo 
fhe currenf solufion. These limifafions regard fhe impossibilify fo ex- 
fend some widely used Java BCL classes (Sfring, Infeger,...) because fhey 
are declared final, and fhe impossibilify fo infercepf all non-privafe class 
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field accesses. 

In the Main class, the programmer just asks to transform the Main 
class and the Mandelbrot ones with PAL, that is, to process the relevant 
PAL annotations and to produce an executable IL which exploits paral¬ 
lelism according to the features (hardware and software) of the target 
architecture where the Main itself is being run. 

4.2.2 Experimental results 

To validate the PAL approach we ran some experiments with the cur¬ 
rent protot 5 rpe we developed. In particular, the conducted experiments 
were aimed at evaluating the effectiveness of PAL approach. It has been 
evaluated measuring the overhead caused by raising the programming 
abstraction by means of PAL. 

We ran tests for each adapter developed, i.e. both for the multithread 
adapter and for the JJPF one. In other words, the tests were covering 
parallel transformations suiting both multiprocessor and cluster archi¬ 
tectures. In the former case, we used, as computing resource for the test¬ 
bed, a h 5 rper-threading bi-processors workstation (Dual Intel Xeon 2Ghz, 
Linux kernel 2.6). In the latter case, instead, we ran the transformed ap¬ 
plication on a blade cluster (24 machines single Pentiumlll-SOOMhz pro¬ 
cessor with multiple Fast Ethernet network, Linux kernel 2.4). In both 
cases, our test application was a fractal image generator, which computes 
sections of the Mandelbrot set. The Mandelbrot set is a set of points in 
the complex plane, the boundary of which forms a fractal. Mathemati¬ 
cally, the Mandelbrot set can be defined as the set of complex c-values 
for which the orbit of 0 under iteration of the complex quadratic poly¬ 
nomial Xn+i = a;„2 -I- c remains bounded. A complex number, c, is in 
the Mandelbrot set if, when starting with a;o = 0 and applying the iter¬ 
ation repeatedly, the absolute value of never exceeds a certain num¬ 
ber (that number depends on c) however large n gets. When computed 
and graphed on the complex plane, the Mandelbrot Set has an elabo¬ 
rate boundary, which does not simplify at any given magnification. This 
qualifies the boundary as a fractal. We picked up Mandelbrot because it 
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PAL-JJPF Efficiency Comparison 



Figure 20: Mandelbrot computation: efficiency comparison with differ¬ 
ent image resolution, processing element number and task computational 
weight. 


is a very popular benchmark for embarrassingly parallel computation. 
PAL addresses exactly these kinds of computations, as it only allows 
executing remotely methods not accessing shared (static) variables nor 
having any kind of side effects. On the one hand, this obviously repre¬ 
sents a limitation, as PAL cannot compete, as an example, with other ap¬ 
proaches supporting plain loop parallelization. On the other hand, huge 
amounts of embarrassingly parallel applications are executed on clus¬ 
ters, workstation networks and grids. Most of times, the implementation 
of these applications requires a significant programming effort, despite 
being "easy" embarrassingly parallel, far more consistent than the effort 
required to execute the same kind of application exploiting PAL. 

To study in more detail the behavior of the transformed, parallel, ver¬ 
sion of the Mandelbrot application in several contexts, we ran the fractal 
generator setting different resolutions (600x400,1200x800 and 2400x1600) 
and task computational weights, starting from 1 up to 40 lines at time. 
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For each test-bed the total number of lines were fixed, hence when fhe 
fask size (number of lines fo compufe) increases, fhe fofal number of 
fasks decreases. 

The Mandelbrof application, when fransformed exploiting fhe mulfi- 
thread adapfer, has been execufed only wifh parDegree paramefer sef fo 
1 or 2 (we used a bi-processor machine for fhe fesf-bed). Neverfheless, 
the multithreaded experiments achieved promising results, as the regis¬ 
tered efficiency wifh parallel degree 2 is very close fo fhe ideal one, for all 
fhe seffing combinations (resolution and compufe lines). Since in a mul- 
ficore solution we have a lower communication impacf fhan in a COW 
or grid solution, we can poinf ouf fhat fhis performance should be easily 
mainfained wifh symmefric multiprocessors even wifh larger (wifh four, 
eighf or more cores) processing elemenfs. 

Affer fhe fesf wifh fhe mulfifhread adapfer, we fesfed also fhe JJPF one 
for disfribufed architecfures. We used fhe very same Mandelbrof source 
code. PAL fransformed if exploiting fhe JJPF adapfer in order fo make 
if able fo be execufed on disfribufed worksfation nefwork. In fhis case, 
we achieved performances definifely close fo fhe ones we achieved wifh 
hand wriffen JJPF code (see Figure p0||. The Figure shows fhe resulf of 
the experiments with an image resolution of 2400x1600 (ofher resulfs ob- 
fained using differenf image resolufions gave comparable resulfs) when 
a differenf number of processing elemenfs are used (i.e. differenf values 
specified fo fhe @Parallel(parDegree=...) annofafion). 

These resulfs demonsfrafe fhaf PAL performance sfricfly depends on 
fhe parallel fool fargefed by fhe PAL IL fransformafion fechniques. Ac- 
fually fhe overhead infroduced by PAL is negligible. 

4.2.3 Learning from PAL experience 

Designing, developing and fhen fesfing PAL we are faughf a lesson by 
exploiting generafive mefaprogramming fechniques coupled wifh pro¬ 
grammers high-level hinfs specified af source code level, if is possible 
fo fransform a java program fhaf own some properfies, enriched wifh 
some proper armofafions, in a parallel program. The parallelization is 
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obtained through the as 5 mchronous and parallel execution of annotated 
methods. Annotated method code is transformed in a macro dafa-flow 
block fhaf can be dispafched fo be execufed on fhe available compufa- 
fional resources. This process execufed af run-fime direcfly af inferme- 
diafe language level, allows fo exploif fhe informafion available fo par¬ 
allelize fhe applications wifh respecf bofh fo fhe parallel fools available 
on fhe fargef execufion environmenf and fo fhe programmer supplied 
non-funcfional requiremenfs. A run-fime fransformafion allows fo hide 
mosf of parallelization issues. The resulfs we obfained are very encour¬ 
aging and show fhaf fhe overhead infroduced by PAL is negligible. Nev- 
erfheless, fhe PAL profof 5 rpe we developed has some limifafions. The 
non-funcfional requiremenfs are limifed fo fhe possibilify fo indicafe fhe 
parallelism degree, fhe paramefer passing semanfic fo PAL-annofafed 
mefhod is limifed fo deep-copy and fhe class fields are nof accessible 
from PAL-annofafed mefhods. Furfhermore, fhe programmer has fo in¬ 
clude an explicif dereferenfiafion of objecfs refurned by PAL-armofafed 
mefhods. Finally, currenf PAL profof 5 rpe allows only very simple forms 
of parallelization. 

In a sense, PAL has been a proof of concepf demonsfrafing fhe effec- 
fiveness of fhe approach. Wifh fhis awareness in mind, we decided fo 
exploif fhe gained experience fo infegrafe some elemenfs of fhe PAL ap¬ 
proach in our modified muskel framework. The goal is fo obfain a frame¬ 
work allowing programmers fo develop customizable parallel sfrucfured 
applications which "parfs" can be fransformed in macro dafa-flow blocks 
optimized af run-fime according fo programmers direcfives and avail¬ 
able hardware and soffware resources. 


4.3 Metaprogramming muskel 

PAL proved fhaf, given fhe exisfence of a proper mefaprogramming run- 
fime supporf, annofafions are a handy way bofh fo indicafe which parfs 
of a program musf run in parallel and fo express non-funcfional require¬ 
menfs direcfly in fhe source code. Such informafion given as inpuf fo 
PAL mefaprogramming engine can be acfually exploited fo opfimize fhe 
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original annotated code with respect to the running platform and the 
programmers' non-functional specifications. Therefore, we decided fo 
apply fhe main feafures of PAL approach fo our modified muskel im- 
plemenfafion. Acfually, adapfing fhem fo muskel we changed a liffle 
bif fhe approach. Such a change is due fo a few mofivafions. Firsf of all 
because muskel provides per se a disfribufed macro dafa-flow execufor 
whereas PAL exploifs exfernal fools for disfribufed program execufion. 
Moreover, we would like fo have a more flexible mechanism for macro 
dafa-flow block generafion and managemenf. Finally, we would like fo 
exploif a sfandard fool for run-fime code fransformafion insfead of using 
ad-hoc fools. As a consequence we decided fo use infegrafe in muskel 
fhe AOP model and in parficular fhe AspecfJ framework. 

The firsf sfep in fhis direcfion was exploifing AspecfJ fo implemenf as- 
pecf driven program normalizafion in muskel. We already infroduced 
normal form and code normalizafion in Secfion 12.2.61 Lef us fo recall 
if briefly. Normalizafion consisfs in fransforming an arbifrary muskel 
program, whose sfrucfure is a generic skeleton free, info a new, equiv- 
alenf one, whose parallel sfrucfure is a farm wifh a worker made up of 
fhe sequenfial composifion of fhe sequenfial skeletons appearing in fhe 
original skeleton free faken leff fo righf. This second program is fhe skele- 
fon program normal form and happens fo perform beffer (wifh respecf 
fo fhe service fime) fhan fhe original one in fhe general case and in fhe 
same way in fhe worsf case. 

As an example, fhe code reported in fhe previous chapfer in Figure 


Skeleton main = new Farm(new Seq{f,g)); 

where Seq is basically a pipeline whose sfages are executed sequenfially 
on a single processor. 

Code normalizafion can be obfained explicifly inserfing sfafemenfs in 
fhe source code. This means fhaf programmers musf change fhe source 
code fo use fhe normal form in place of fhe non-normal form version of 
fhe same program. Exploifing AspecfJ we defined a proper aspecf deal¬ 
ing wifh normal form fransformafion by defining a poinfcuf on fhe exe- 


11 can be fransformed info fhe equivalenf normal form code: 
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public aspect Normalize { 

public boolean ContractSpecified = false; 
public boolean normalized = false; 

// contract is an integer, to simplify ... 
public int contract = 0; 

pointcut calledContract(int i): call(public void Manager.setContract(int)) && args(i); 

void around{int i): calledContract(i){ 

ContractSpecified = true; 
contract = i; 
proceed(i); 

} 

pointcut callSetProgram(Skeleton c): call(public void Manager.setProgram(Skeleton)) && args(c); 

void around{Skeleton c): 
callSetProgram(c) { 
normalized = true; 
proceed(new NormalForm(c)); 

} 

pointcut callEval(Manager m) : call(public void Manager.evalQ) && target(m); 

before(Manager m):callEval{m){ 
if(ContractSpecified) 
if(normalized) 

m.setContract{Manager.NormalizeContract(contract)); 

else 

m.setContract(Manager.DefaultNormalizedContract); 

} 

} 

} 


Figure 21: Aspect} code handling performance contracts in muskel. 


cution of the setProgram Manager method and associating to the point- 
cut the action performing normal form transformation on the source code 
in the aspect, such as the one of Figure As a consequence, the pro¬ 
grammers can decide whether to use the original or the normal form ver¬ 
sion of the program just picking up the standard Java compiler or the As¬ 
pect} one. The fact the program is left unchanged means the programmer 
may debug the original bug and have the normal form one debugged too 
as a consequence, provided the AOP code in the normal form aspect is 
correct. Moreover, exploiting aspects as discussed above, we handled 
also related features by means of proper aspects. In fact, in case the pro- 
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public aspect Normalize { 

pointcut callSetProgram(Skeleton c): 

call{public void Manager.setProgram(Skeleton)) && args(c): 


void around(Skeleton c) 

: callSetProgram(c) { 

proceed(new NormalForm(c)); 

} 


Figure 22: AspectJ code modeling normal form in muskel. 


grammer provided a performance contract (a parallelism degree, in the 
simpler case) and then used the AspectJ compiler to ask normal form 
execution of the program, it turns out to be quite natural imagine a fur¬ 
ther aspect handling the performance contract consequently. Figure 
shows the AspectJ aspect handling this aspect. In this case, contracts are 
stored as soon as they have been issued by the programmer, with the first 
pointcut, then, in when normalization has been required (second point- 
cut) and program parallel evaluation is required, the contract is handled 
consequently (third pointcut), that is, it is either left unchanged or a new 
contract is derived from the original one according to some normal form 
related procedure. 

The second step consisted in testing the integration of muskel with 
AspectJ to in a more complex scenario. Hence, we exploited the aspect 
oriented programming support integrated in muskel in order to de¬ 
velop workflows which structure and processing are optimized at run¬ 
time. 


4.4 Workflows with muskel 

Workflows represents a popular programming model for grid applica¬ 
tions (l74t . In a workflow, programmers express the data dependencies 
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that incurs among a set of blocks, possibly using a DAG. Each block pro¬ 
cesses input data to produce output data. Workflow schedulers arrange 
fhe compufafions for grid execution in such a way 

• all the parallelism implicitly defined fhrough fhe (absence of) de¬ 
pendencies in fhe DAG is exploifed, and 

• available grid resources (processing elemenfs) are efficienfly used. 

In a sense, a programming model fhaf eases fhe developmenf of efficienf 
workflow applications can be successfully exploifed for fhe developmenf 
of many grid applications. For fhis reason, we conceived an approach 
aimed at the implementation of workflows on fop of fhe muskel dis- 
fribufed macro dafa-flow inferprefer. We fook into account the execution 
of workflows on a sef of inpuf dafa ifems. The sef of inpuf dafa ifems rep- 
resenfs fhe program inpuf sfream. Each ifem on fhat sfream will be sub¬ 
mitted fo a full workflow compulation. The resulfs of fhaf compulation 
will appear as a dafa ifems onto fhe program outpuf sfream. Usually fhe 
workflows considered in grids are made of nodes fhaf are compufafion- 
ally complex. Possibly parallel applications processing dafa confained 
in one or more inpuf files fo produce dafa in one or more oufpuf files 
m- We considered a very simple class of workflows: fhose whose DAG 
nodes are Java "functions" processing a generic Objecf inpuf parameters 
fo produce an Objecf oufpuf resulfs. 

4.4.1 Aspects to implement workflows 

As already slated, we considered workflows processing sfream of inpuf 
dafa fo produce sfream of oufpuf dafa. Acfually, fhese are nof classical 
workflows. As discussed in fhe following, however, classical workflows 
can be efficienfly addressed as well as a side effecl of fhe efficienf im- 
plemenfation of sfream parallel workflows. This allows fo express bofh 
parallelism implicif in fhe workflow definition (and fherefore exploifed 
wifhin fhe compulation of a single insfance of fhe workflow) and sfream 
parallelism (parallelism among disfincf insfances of workflow compu¬ 
lation, relative fo independenf inpuf dafa ifems). In order fo obfain a 
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macro data-flow graph from the workflow abstract code, we exploited 
the AspectJ AOP framework ((84): 

• Programmers express workflows as plain Java code, with the con¬ 
straint the nodes of the workflow must be expressed using Com¬ 
pute object calls. 

• Programmers declare a Manager object passing it an Iterator pro¬ 
viding the input tasks. The Manager object completely and trans¬ 
parently takes care of implementing stream parallelism using the 
muskel distributed macro data-flow interpreter. 

• AOP pointcuts and advices are used to intercept the calls to the 
compute methods and to transform such calls into proper fireable 
macro data-flow instructions submitted to the muskel distributed 
data-flow interpreter. 

Sample code used to model workflows is shown in Figure The 
right part of the Figure lists the Java code modeling the workflow graph¬ 
ically depicted in the left part of the Figure. Multiple results are mod¬ 
eled returning Vector objects and multiple input parameters are modeled 
with a "vararg" compute method^. 



Figure 23: Sample workflow (left) and relative Java code (right) 


^ varargs have been introduced in Java 1.5 and allow to pass a variable number of argu¬ 
ments (of the same type) to a method; the arguments are referred to in the method body as 
array elements 
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More in detail, the calls to compute methods are transformed into 
the submission of a proper (already fireable) macro dafa-flow insfrucfion 
fo fhe muskel disfribufed macro dafa-flow inferprefer modified in such 
a way a Future for fhe resulf is immediafely refumed. If one of fhe in- 
puf argumenfs of fhe compute call is a Future, the advice intercepting 
the compute method call takes care of waifing for ifs acfual value fo be 
compufed before submiffing fhe macro dafa-flow insfrucfion fo fhe infer¬ 
prefer. 

As inpuf Future actual values are only required by the advice right 
before fhe workflow node is sfarfed, parallelism implicif in fhe workflow 
is correcfly delegafed fo fhe underlying muskel inferprefer. As an ex¬ 
ample, consider fhe workflow of Figure The funcfions G1 and G2 
are evaluafed (fheir evaluation is requested by the advice to muskel in¬ 
terpreter) sequentially. However, as the first one immediately returns a 
Future, the second one (also returning a Future) will eventually run in 
parallel on a distinct remote processing element as outlined in Figure]^ 
When the evaluation of fhe H node is requesfed, fhe advice infercepf- 
ing fhe requesf will realize fwo fufures are passed as inpuf paramefers 
and fherefore if will waif before submiffing fhe node evaluafion requesf 
fo fhe muskel inferprefer up fo fhe momenf fhe fwo acfual values of 
fhe "inpuf" Futures are available. Overall, advices transforming calls fo 
compute methods into fireable macro dafa-flow insfrucfions acf as fhe 
dafa-flow matching unit, according fo classical dafa-flow jargon. 

The approach suggesfed here fo implemenf workflows on fop of fhe 
muskel macro dafa-flow inferprefer presenfs af leasf fwo significanf ad- 
vanfages: 

• fhe whole, already exisfing, efficienf and assessed muskel macro 
dafa-flow inferprefer sfrucfure is fully exploded. The muskel in¬ 
ferprefer fakes complefely care of ensuring load balancing, faulf 
tolerance (w.r.f. remofe resource faulfs) and securify; 

• programmers are only asked fo express workflows with elemen¬ 
tary Java code, possibly spending some time wrapping workflow 
node code in Compute objects and declaring a Manager object which 
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Figure 24: Transition diagram relative to the execution of part of the work- 
flow of Figure [23| 


is used to supply input data, retrieve output data, control non func¬ 
tional features (e.g. parallelism degree in the execution of fhe work- 
flow) and fo ask fhe evaluation of fhe workflow code. 

• As in PAL, fransformafion can be easily disabled. This means fhaf 
fhe programmers' applicafion code can be run fhrough a classical 
sequential compiler/inferprefer suife and debugged using normal 
debugging fools. 
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4.4.2 Aspects with muskel: implementation details 

In order to be able to express workflows, the programmer must write 
one class per workflow node. The class has fo implemenf fhe Compute 
interface, which is a very simple inferface such as: 


public interface Compute extends Serializable{ 
public Object compute(Object... params); 

} 


The compute mefhod is assumed fo compufe fhe workflow node re- 
sulfs (fhe refurned Object) out of fhe inpuf paramefers params. Then 
fhe workflow can be described in a class implemenfing fhe Workflow 
inferface, which is defined as follows: 


public interface Workflow { 

public Object doWorkflow(Object param); 

} 


As an example, a workflow can be described by fhe class: 


public class WorkFlowl implements Workflow { 
public Object doWorkflow(Object task) { 

Vector resF = (Vector) F.compute(((Vector)task).elementAt(0)); 
Object resGl = G1.compute(resF.elementAt(0)); 

Object resG2 = G2.compute( resF.elementAt(1), 

((Vector)task).elementAt(1) ) ; 

Object resH = H. compute(resGl, resG2); 
return resH; 



The code sfyle here is quife close fo fhe sfyle used when programming 
plain Java applications. 

We capfure fhe execution of fhe Compute calls in fhe workflow ex¬ 
ploiting aspecfs. The poinfcuf is defined on fhe calls of fhe compute 
mefhod of any objecf implemenfing Compute: 


pointcut computeRemotely(Object param[], itfs.Compute code) : 
call(Object itfs.Compute.compute(Object ... )) && 

!within(execEngine.Engine) && 
args(param) && target(code) ; 


The advice invoked on fhe poinfcuf is an around advice such as: 
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execEngine.Engine eng = new execEngine.Engine{); 


Future around(Object param[], itfs.Compute code) 
:computeRemotely(param, code) { 
for(int i=0; i<param.length; i++) { 

// reifing each parameter right before call 
if{param[i] instanceof Future) { 

param[i] = ((Future) param[i]).getValue(); 


// deliver fireable instruction 
Object future = eng.exec(codice, param); 

// and return the corresponding Future object 
return future; 


It arranges to collect the Compute class name and the input parame¬ 
ters and creates a macro data-flow instruction, which is submitted to the 
distributed muskel macro data-flow inferprefer via fhe predefined exe¬ 
cEngine objecf insfance declared in fhe aspecf class. Inpuf tokens fo fhe 
macro dafa-flow insfrucfion fhaf are Future instances rather than plain 
reified objecfs, are evenfually reified on the fly wifhin fhe advice. Evenfu- 
ally, a Future object is returned. It can be eventually used to retrieve the 
actual data computed by the distributed interpreter during the compute 
call. In particular. Future interface provides two mefhods: a getValueO 
mefhod fo gef fhe acfual value of fhe Future, possibly waiting for fhe 
completion of fhe corresponding compufafion, and a boolean isReadyO 
mefhod fo fesf whefher fhe compufafion producing fhe acfual value of 
fhe Future is already terminated 

As a whole, the procedure just described models an as 5 mchronous ex¬ 
ecution of fhe macro dafa-flow insfrucfions implementing fhe workflow 
nodes. If allows fo fully exploif fhe parallelism infrinsic fo fhe workflow, 
by properly using Futures. 

As already stated, we are interested not only in the exploitation of 
parallelism wifhin fhe evaluation of a single workflow insfance, buf also 
in exploifing fhe parallelism among differenf insfances of workflows run 
on disfincf inpuf dafa sefs. In order fo supporf sfream parallelism, we 
provide fhe programmer wifh a Streamiterator manager. This manager 
fakes as parameters an Iterator (providing the input data sets to be pro¬ 
cessed by the Workflow) and a Workflow. It provides a method to com- 
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pute the whole bunch of inputs, as well as a method to get an Iterator that 
can be used to retrieve workflow results. Using the Streamiterator man¬ 
ager, the main code relative to our example can therefore be expressed 
as follows: 

public Static void main(String[] args) { 

// workflow to be used (userdef) 

Workflow wf = new WorkFlowl(); 

// provide the input tasks via an iterator (userdef) 

InTaskIterator intIt = 

new InTaskIterator0; 

// declare the manager 

Manager mgr = new Streamiterator(wf,intIt); 

// start parallel computation 
mgr.go 0; 

// get access to result iterator 
Iterator resit = mgr.getResultIterator(); 

// while there are more results ... 
while(resit.hasNext()) { 

// get one and 

Object result = resit.next (); 

// process it (userdef) 


The main task of the Streamiterator manager is to invoke execution 
of the parameter Workflow instances on all the input data sets provided 
by the Iterator. This is achieved exploiting a proper Thread pool and 
activating one thread in the pool for each independent workflow com¬ 
putation. Then, the AOP procedure illustrated above intercepts the calls 
to compute methods and arrange to run them in parallel through the 
muskel distributed macro data-flow interpreter. 

4.4.3 Experiments 

In order to prove the effectiveness of the approach, we tested it making 
some experiments on a distributed computing architecture (a network 
of workstations, actually). We directly used Java (version 1.5) accessible 
via plain secure shell (ssh/scp) rather than with other more sophisticated 
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Figure 25: Efficiency of the muskel/aspect workflow prototype 


grid middleware. It is worth to point out that the tests have not been 
conducted to evaluate the scalability of plain muskel, that has actu¬ 
ally already been demonstrated, as discussed in (l58t . Rather, the tests 
have been performed in order to give an estimation of the overhead in¬ 
troduced by aspect] transformations. 

In fact, the only difference between plain muskel and the system 
proposed here, able to execute workflows on top of muskel, lies in the 
way the fireable instructions are provided to the distributed data-flow 
interpreter of muskel. Actually, in plain muskel, fireable instructions 
are retrieved from a compiled representation of a data-flow graph. In 
particular, each time a new token arrives to a macro data-flow instruc¬ 
tion in the graph (either from the input stream or as the result of the dis¬ 
tributed computation of another macro data-flow instruction) the target 
data-flow instruction is checked for "fireability" and, possibly, delivered 
to the distributed macro data-flow interpreter. The time spent is in the 
sub-micro second range (only considering net time, not taking into ac- 


118 

























count time spent to copy parameters in memory during the interpreter 
call). When executing workflows according to the approach discussed 
here, instead, flreable instructions are generated by means of the aspect] 
tool. In particular, they come from fhe "advice" invoked on fhe "poinf- 
cuf" infercepfing fhe compute calls. In order to estimate the overhead 
introduced by using these Aspect Oriented Techniques we measured the 
time spent to intercept the compute calls and to transform fhem in macro 
dafa-flow blocks. The measuremenf resulfs are shown in fhe following 
fable (times are in milliseconds): 


Average 

23.09 

Minimum 

19 

Standard deviation 

3.01 

Maximum 

27 


These values are relafive fo an Intel dual-core machine (2 GHz Core 2 
Duo machine), rurming Mac OS/X 10.4, Java 1.5.0_07, AspectJ 1.5.4 with 
AspectJ tools 1.4.2 and Eclipse 3.2.2. On the same machine, deliver¬ 
ing a flreable instruction to the macro data-flow interpreter with plain 
muskel requires a time average of 0.004 milliseconds. The difference in 
fhe times is nof surprising: in fhe former case, we go fhrough pure mefa 
programming fools and we "inferpref" each call, while in fhe latter we 
use plain (compiled) Java fo handle each one of fhe calls. 

Therefore, we can conclude fhe average 23 milliseconds represent the 
pure overhead spent each time a new flreable instruction has to be com¬ 
puted (i.e. each time one of fhe workflow Compute nodes is computed). 
The time spent in Future reification (i.e. filling fhe objecf placeholder 
wifh fhe compufed value, once available), insfead, is negligible (fhis nof 
faking info accounf fhe time spenf fo waif for acfual producfion of Future 
values, of course). This allows us fo conclude fhaf fhe parallel execution 
of workflows on fop of muskel slighfly increases fhe grain required fo 
achieve almosf ideal scalabilify. 

In facf. Figure shows how wifh suifable grain of fhe workflow 
nodes (i.e. of fhe Compute functions) efficiency close fo fhe ideal one 
is achieved. 
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4.5 Differences between the two approaches 


As we already stated before, both PAL and AspectJ enriched muskel 
(AEM) were conceived, designed and implemented to provide a proof- 
of-concept of our mefaprogramming approach fo sfrucfured parallel pro¬ 
gramming. Acfually, fhey enforce code parallelizafion via a hinfs-driven 
code fransformafion. Hinfs are provided by programmers in fhe form of 
java annofafions (PAL) and AspecfJ rules AEM. Even if fhe fwo frame¬ 
works affain fhe same idea, fhey are slighfly differenf. The main differ¬ 
ences befween fhe fwo frameworks are: 

• In AEM fhere is a sharp-cuf disfincfion befween fhe "confrol" and 
"business" code, acfually confained in separafe files, whereas wifh 
PAL programmers wrife business code and annofafions (fhaf be¬ 
haves as confrol code) inside fhe same file. 

• PAL was conceived fo exploif mefhod-level parallelism: fhrough 
a simple program enrichmenf process, programmers choose which 
Java mefhods-call should be fransformed in as 5 mchronous ones, i.e. 
PAL allows fo add parallelism fo legacy java code wifh a minimal 
infervenfion. Insfead, in AEM programmers have fo implemenf 
fheir applicafion as a workflow. 

• PAL provides a fixed number of annofafions (hence a very limifed 
number of action can be performed) fhaf an adapfer-based archi- 
fecfure exploifs fo fransform byfecode af run-fime. The fransfor¬ 
mafion process depends, in a way, on fhe adapfer used. In AEM 
fhe code fransformafion policies implemenfafion is based on As¬ 
pecfJ, fhe mosf widely diffused fool for aspecf orienfed program¬ 
ming, which offers a rich sef of mechanisms for cusfomizing fhe 
"aspecfizafion" process. As a consequence, fhe programmers can 
customize/opfimize/change fhe fransformafion process by simply 
modifying fhe aspecfs (wifhouf a direcf code updafe). 
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Summarizing the Chapter 


In this Chapter we presented two results, about the exploitation of metapro¬ 
gramming techniques in structured parallel programming environment. ]Ne 
exploited those techniques in order to generate and optimize at run-time macro 
data-flow blocks without directly dealing with their low-level management. First 
we presented a new technique for high-level parallel programming based on the 
introduction of a Parallel Abstraction Layer (PAL). PAL does not introduce a 
new parallel programming model, but actually exploits the programmer knowl¬ 
edge provided through annotations to restructure at run-time the application, 
hiding most of parallelization issues, once it notice the information about the 
running platform. This process is executed directly at intermediate language 
level. This allows to have a portable code transformation mechanism without 
paying a complete code recompilation for each change in the code. In order to 
have a proof-of-concept of the approach we developed a PAL Java prototype and 
we used it to perform some experiments. The results are very encouraging and 
show that the overhead introduced by PAL is negligible, while keeping the pro¬ 
grammer effort to parallelize the code negligible. Then we presented the other re¬ 
sult we obtained integrating the Aspect]framework with our modified muskel. 
We described how AOP techniques can be seamlessly used to transform a very 
basic kind of workflows in such a way they can be executed on distributed tar¬ 
get architectures through the muskel macro data-flow interpreter. How AOP 
techniques allow to completely separate the concerns relative to parallelism ex¬ 
ploitation and application functional core. In particular, the same application 
code used to perform functional debugging on a single, sequential machine may 
be turned into parallel code by adding aspects, compiling it through Aspect] 
and then running it on the muskel run-time support. The way used to write 
workflow code is quite basic Java programming. Workflow components must im¬ 
plement a simple interface, and programmers are explicitly required to provide 
them as side effect free sequential components. The experiments conducted show 
that the approach is perfectly feasible and that actual speedups can be achieved 
provided that the workflow nodes are medium to coarse grain. 
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Chapter 5 

Behavioural Skeletons 


Chapter road-map In this chapter we present Behavioural Skeletons, an ap¬ 
proach, we contribute to conceive and validate, aimed at providing programmers 
with the ability to implement autonomic grid component-based applications that 
completely take care of the parallelism exploitation details by simply instantiat¬ 
ing existing skeletons and by providing suitable, functional parameters. The 
model has been specifically conceived to enable code reuse and dynamicity han¬ 
dling. ]Ne start describing (Section 5.1 • how component-based application can 
ease the task of developing grid applications. Then we outline the Grid Com¬ 
ponent Model (Section 5.21 with respect to its autonomic features. After we 
present the Behavioural Skeletons model (Section 5.4( a set of noteworthy Be¬ 
havioural Skeletons (Section 5.5 ' and their implementation (Section 5.6 L At 
the end of chapter we describe a set of experiment we conducted to validate the 
Behavioural Skeletons model (Section \5y\ . 


5.1 Components to simplify Grid programming 

Developing grid applications is even more difficult than programming 
traditional parallel applications. This is due to several factors as, the 
heterogeneity of resources, their worldwide distribution, their d 5 mamic 
recruiting and releasing. Indeed, when programming Grid applications 
neither the target platforms nor their status are fixed (l82t . 
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As a consequence, grid applications need to d 5 mamically adapt to 
the features of fhe underlying archifecfure in order fo be efficienf and/or 
high performance (IT9t . In recenf years, several research inifiafives ex¬ 
ploiting componenf fechnology (f52ll have invesfigafed fhe area of com- 
ponenf adapfafion, i.e. fhe process of changing fhe componenf for use in 
differenf confexfs. This process can be eifher sfafic or d 5 mamic. 

The basic use of sfatic adapfation covers sfraighfforward buf popular 
mefhodologies, such as copy-paste, and OO inheritance. A more advanced 
usage covers fhe case in which adapfafion happens af run-fime. These 
sysfems enable d 5 mamically defined adapfafion by allowing adapfafions, 
in fhe form of code, scripfs or rules, fo be added, removed or modified 
af run-fime (l37l . Among fhem is worfh fo disfinguish fhe sysfems where 
all possible adapfafion cases have been specified af compile-fime, buf fhe 
conditions defermining fhe acfual adapfafion af any poinf in time can 
be d 5 mamically changed (|23). Dynamically adapfable sysfems rely on 
a clear separation of concerns between adapfafion and applicafion logic. 
This approach has recenfly gained increased impefus in fhe grid commu¬ 
nity, especially via its formalization in terms of fhe Autonomic Computing 
(AC) paradigm (l22tl24l[77t . The AC ferm is emblematic of a vasf hierarchy 
of self-governing sysfems, many of which consisf of many inferacfing, 
self-governing componenfs fhaf in fum comprise a number of inferacf¬ 
ing, self-governing componenfs at the next level down (l83t . An auto¬ 
nomic component will typically consist of one or more managed compo¬ 
nenfs coupled with a single autonomic manager that controls them. To 
pursue its goal, the manager may trigger an adaptation of fhe managed 
componenfs fo reacf fo a run-fime change of applicafion QoS require- 
menfs or fo fhe plafform sfafus. 

In fhis regard, an assembly of self-managed componenfs implemenfs, 
via fheir managers, a disfribufed algorifhm fhaf manages fhe entire ap¬ 
plication. Several existing programming frameworks aim fo ease fhis 
fask by providing a sef of mechanisms fo d 5 mamically insfall reacfive 
rules within autonomic managers. These rules are t 5 rpically specified 
as a collection of when-ez^enf-if- cond-then-flcf clauses, where event 
is raised by fhe monitoring of componenf infernal or exfernal acfivify 
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(e.g. the component server interface received a request, and the platform 
rurming a componenf exceeded a fhreshold load, respecfively); cond is 
an expression over componenf infernal affribufes (e.g. componenf life- 
cycle sfafus); act represenfs an adapfafion action (e.g. creafe, desfroy a 
componenf, wire, unwire componenfs, nofify evenfs fo anofher compo¬ 
nenf's manager). Several programming frameworks implemenf varianfs 
of fhis general idea, including ASSIST ([19llll3> , AufoMafe (l93l , SAFRAN 
( [651 , and finally fhe forfhcoming CoreGrid Componenf Model (GCM) 
((52). The latter two are derived from a common ancesfor, i.e. fhe Fracfal 
hierarchical componenf model (139) . All fhe named frameworks, excepf 
SAFRAN, are fargefed fo disfribufed applicafions on grids. 

Though such programming frameworks considerably ease fhe devel- 
opmenf of an aufonomic application for fhe grid (fo various degrees), 
fhey rely fully on fhe applicafion programmer's experfise for fhe sef-up 
of fhe managemenf code, which can be quife difficulf fo wrife since if 
may involve fhe managemenf of black-box componenfs, and, nofably, is 
failored for fhe particular componenf or assembly of fhem. As a resulf, 
fhe infroducfion of dynamic adapfivify and self-managemenf mighf en¬ 
able fhe managemenf of grid d 5 mamism, and uncerfainfy aspecfs buf, af 
fhe same fime, decreases fhe componenf reuse pofenfial since if furfher 
specializes componenfs wifh applicafion specific managemenf code. 

From fhe poinf of view of issues fo address for designing and devel¬ 
oping nexf generafion sfrucfured parallel programming sysfems, fhis is 
a big problem. Indeed, if on fhe one hand making componenfs adapfive 
addresses fhe issue of handling d 5 mamicify (issue number VII), on fhe 
ofher hand if impairs fhe code reuse (issue number V). In fhis chapfer we 
cope wifh fhis problem proposing Behavioural Skeletons as a novel way fo 
describe aufonomic componenfs in fhe GCM framework. We confribufed 
significanfly fo fheir conception, design and implemenfafion fogefher 
wifh ofher researchers, co-aufhored of fhe papers dltil flT) in which we 
presenfed fhis model. My personal confribufion has mainly concerned 
fhe definifion of fhe fask farm Behavioural Skelefon as well as fhe imple¬ 
menfafion of fhaf skelefon wifhin GridCOMP. 

Behavioural Skelefons aim fo describe recurring pafferns of compo- 
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nent assemblies that can be (either statically or d 3 mamically) equipped 
with correct and effective management strategies with respect to a given 
management goal. Behavioural Skeletons help the application designer 
to i) design component assemblies that can be effectively reused, and ii) 
cope with management complexity by providing a component with an 
explicit context with respect to top-down design (i.e. component nest¬ 
ing)- 


5.2 GCM: the Grid Component Model 

GCM is a hierarchical component model explicitly designed to support 
component-based autonomic applications in highly d 3 mamic and hetero¬ 
geneous distributed platforms, such as grids. If is currenfly under devel- 
opmenf by fhe parfners of fhe EU CoreGRID Network of Excellence^. A 
companion EU STREP projecf, GridCOMP ^ is going fo complefe fhe de- 
velopmenf of an open source implemenfafion of GCM (preliminary ver¬ 
sions are already available for download as embedded modules in fhe 
ProAcfive middleware suife)^. GCM builds on fhe Pracfal componenf 
model (l39ll and exhibifs fhree prominenf feafures: hierarchical composi- 
fion, collecfive inferacfions and aufonomic management. We participate 
to both the projects (CoreGrid & GridComp) and collaborate for fhe de¬ 
sign and developmenf of GCM, in parficular in fhe contexf of aufonomic 
management. The full specificafion of GCM can be found in ([5^ . 

Hierarchical composition As in fractal, a GCM component is composed 
of two main parts: the membrane and the content. The membrane is an ab¬ 
stract entity that embodies the control behavior associated with a compo¬ 
nent, including the mediation of incoming and outgoing invocations of 
content entities. The content may include either the code directly imple¬ 
menting functional component behavior (primitive) or other components 
(composite). In the latter case, the included components are referred as 

^http://www.coregrid.net 

^http://gridcomp.ercim.org 

^http://www-sop.inria.fr/oasis/ProActive 
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the inner components. GCM components, as Fractal ones, can be hierar¬ 
chically nested to any level. Component nesting represents the imple- 
mentedJ}y relationship. Composite components are first class citizens 
in GCM and, once designed and implemented, they carmot be distin¬ 
guished from primifive, non-composife ones. 

Collective interactions The Grid Component Model allows component 
interactions to take place with several distinct mechanisms. In addi¬ 
tion to classical "RPC-like" use/provide ports (or client/server inter¬ 
faces), GCM allows dafa, sfream and even! porfs fo be used in com- 
ponenf inferacfion. Bofh sfafic and dynamic wiring between dual in- 
f erf aces is supporfed. Each inferface may expose several operations of 
differenf fypes. Furfhermore, collecfive inferacfion pafferns (communi- 
cafion mechanisms) are also supporfed. In parficular, composife com- 
ponenfs may benefif from customizable one-fo-many and many-fo-one 
functional inferfaces fo disfribufe requesfs arriving fo one componenf's 
porf fo many irmer componenfs and gafher requesfs from many irmer 
componenfs fo a single oufgoing porf. 

Autonomic management Autonomic management aims to attack the 
complexity which entangles the management of complex sysfems (as ap¬ 
plications for Grids are) by equipping fheir parfs wifh self-managemenf 
facilifies ( l83] |. GCM is fherefore assumed fo provide several levels of au- 
fonomic managers in componenfs, fhaf fake care of fhe non-funcfional 
feafures of fhe component programs. GCM components thus have two 
kinds of inferfaces: functional and non-funcfional ones. The functional 
inferfaces hosf all fhose porfs concerned with implementation of fhe func¬ 
tional features of fhe componenf. The non-funcfional inferfaces hosf all 
fhose porfs needed fo support the component management activity in 
the implementation of fhe non-funcfional feafures, i.e. all fhose feafures 
confribufing fo fhe efficiency of fhe component in obtaining the expected 
(functional) results but not directly involved in result computation. Each 
GCM component therefore confains an Autonomic Manager (AM), infer- 
acfing wifh ofher managers in ofher componenfs via fhe componenf non- 
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functional interfaces. The AM implements the autonomic cycle via a 
simple program based on the reactive rules described above. In this, 
the AM leverages on component controllers for the event monitoring and 
the execution of reconfiguration actions. In GCM, the latter controller is 
called the Autonomic Behaviour Controller (ABC). This controller exposes 
server-only non-functional interfaces, which can be accessed either from 
the AM or an external component that logically surrogates the AM strat¬ 
egy. From the point of view of autonomic features, the GCM components 
exhibiting just the ABC are called passive, whereas the GCM components 
exhibiting both the ABC and the AM are called active. 

5.3 Describing Adaptive Applications 

The architecture of a component-based application is usually described 
via an ADL (Architecture Description Language) text, which enumer¬ 
ates the components and describes their relationships via the used-by re¬ 
lationship. In a hierarchical component model, such as the GCM, the 
ADL describes also the implemented-by relationship, which represents the 
component nesting. 

However, the ADL supplies a static vision of an application, which 
is not fully satisfactory for an application exhibiting autonomic behavior 
since it may autonomously change behavior during its execution. Such 
change may be of several t 5 rpes: 

• Component lifecycle. Components can be started or stopped. 

• Component relationships. The used-by and /or implemented-by rela¬ 
tionships among components are changed. This may involve com¬ 
ponent creation/destruction, and component wiring alteration. 

• Component attributes. A refinement of the behavior of some com¬ 
ponents (which does not involve structural changes) is required, 
usually over a pre-determined parametric functionality. 

In the most general case, an autonomic application may evolve along 
adaption steps that involve one or more changes belonging to these three 
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classes. In this regard, the ADL just represents a snapshot of the launch 
time configuration. 

The evolution of a componenf is driven by ifs AM, which may requesf 
managemenf acfion wifh fhe AM af fhe nexf level up in order fo deal 
wifh managemenf issues if cannof solve locally. Overall, if is a parf of a 
disfribufed sysfem fhaf cooperafively manages fhe enfire applicafion. 

In fhe general case, fhe managemenf code executing in fhe AM of a 
componenf depends bofh on fhe componenf's funcfional behavior and 
on fhe goal of fhe managemenf. The AM should also be able fo cooper- 
afe wifh ofher AMs, which are unknown af design fime due fo fhe na¬ 
ture of componenf-based design. Currenfly programming frameworks 
supporting the AC paradigm (such as the ones mentioned in Section 
|5.1| just provide mechanisms to implement management code. This ap¬ 
proach has several disadvantages, especially when applied to a hierar¬ 
chical component model: 

• The management code is difficult to develop and to test since the 
context in which it should work may be unknown. 

• The management code is tailored to the particular instance of fhe 
managemenf elemenfs (inner componenfs), furfher resfricfing fhe 
componenf reusabilify possible. 

5.4 Behavioural Skeletons 

Behavioural Skeletons aim fo absfracf paramefric paradigms of fhe GCM 
componenfs assembly, each of fhem specialized fo solve one or more 
managemenf goals belonging fo fhe classical AC classes, i.e. configu¬ 
ration, opfimization, healing and profecfion. 

They represenf a specializafion of fhe algorithmic skeleton concept 
for component management. Behavioural Skeletons, as algorithmic skele¬ 
tons, represent patterns of parallel compulations (which are expressed in 
GCM as graphs of componenfs), buf in addition fhey exploif skelefons' 
inherenf semanfics fo design sound self-managemenf schemes of parallel 
componenfs. 
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As a byproduct. Behavioural Skeletons allow categorization of GCM 
designers and programmers into three classes. They are, in increasing 
degree of expertise and decreasing cardinalify: 

1. GCM users: fhey use Behavioural Skelefons fogefher wifh fheir pre¬ 
defined AM sfrafegy. In many cases fhey should jusf insfanfiafe 
a skelefon wifh inner componenfs, and gef as resulf a composife 
componenf exhibiting one or more self-managemenf behaviors. 

2. GCM expert users: fhey use Behavioural Skelefons overriding fhe 
AM management strategy. However, the specialization does not in¬ 
volve the ABC and thus does not require specific knowledge abouf 
fhe GCM membrane implemenfafion. 

3. GCM skeleton designers: fhey infroduce new Behavioural Skelefons 
or classes of fhem. To fhis end, fhe design and developmenf of a 
brand new ABC mighf be required. This may involve fhe definifion 
of new inferfaces for fhe ABC, fhe implemenfafion of fhe ABC ifself, 
fogefher wifh ifs wiring wifh ofher confrollers, and fhe design and 
wiring of new infercepfors. Obviously, fhis requires quife a deep 
knowledge of fhe parficular GCM implemenfafion. 

Due fo fhe hierarchical nafure of GCM, Behavioural Skelefons can be 
identified wifh a composife component with no loss of generalify (iden- 
fifying skelefons as parficular higher-order componenfs (jZS)). 

Since skelefons are fully-fledged GCM componenfs, fhey can be wired 
and nesfed via sfandard GCM mechanisms. From fhe implemenfafion 
viewpoinf, a Behavioural Skelefon is a partially defined composife com¬ 
ponenf, i.e. a componenf wifh placeholders, which may be used fo in¬ 
sfanfiafe fhe skelefon. As skefched in Figure]^ fhere are fhree classes of 
placeholders: 

1. The functional inferfaces S and C fhaf are GCM membrane con¬ 
frollers (fhus objecfs). 

2. The AM fhaf is a parficular inner componenf. If includes fhe man- 
agemenf plan, ifs goal, and exporfed non-funcfional inferfaces. 
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3. Inner component W, implementing the functional behavior. 

The orchestration of fhe inner componenfs is implicifly defined by fhe 
skeleton t 5 rpe. In order fo insfanfiafe fhe skeleton, placeholders should 
be filled wifh suifable enfifies. Observe fhaf jusf enfifies in fhe former fwo 
classes are skeleton specific. Indeed, fhe placeholders of fhe fhird class, 
representing fhe inner componenfs implemenfing fhe functional behav¬ 
ior, are filled wifh user-defined componenfs. The enfifies parf of fhe firsf 
fwo classes characferize fhe composife componenf as a higher order one 
orchesfrafing fhe enfifies of fhe fhird class; like fradifional skeletons are 
higher order funcfions faking as parameter user specified funcfions. 

Behavioural Skelefons usage helps designers in fwo main ways. Firsf, 
fhe applicafion designer benefifs from a library of skelefons, each of fhem 
carrying several pre-defined, efficienf self-managemenf sfrafegies. Then, 
fhe componenf/applicafion designer is provided wifh a framework fhaf 
helps bofh fhe design of new skelefons and fheir implemenfafion. 

In bofh cases fwo feafures of Behavioural Skelefons are exploited: on 
fhe one hand, fhe skelefons exhibif an explicif higher-order funcfional se¬ 
mantics fhaf delimifs fhe skeleton usage and definition domain. On fhe 
ofher hand, fhe skelefons describe paramefric inferacfion pafferns and 
can be designed in such a way fhaf parameters affecf non-funcfional be¬ 
havior buf are invarianf for funcfional behavior. 


5.5 A Basic Set of Behavioural Skeletons 

Here we presenf a basic sef of Behavioural Skelefons. Despite fheir sim- 
plicify, fhey cover a significanf sef of parallel compulations of common 
usage. 

The presenfed Behavioural Skelefons springs from fhe idea of func¬ 
tional replication. Lef us assume fhese skelefons have fwo funcfional in- 
ferfaces: a one-fo-many sfream server S, and a many-fo-one clienf sfream 
inferface C (see Figure [2^. The skelefon accepfs requesfs on fhe server 
inferface; and dispafches fhem fo a number of insfances of an inner com¬ 
ponenf W, which may propagate resulfs oufside fhe skelefon via C infer- 
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face. Assume that replicas of W can safely lose the internal state between 
different calls. For example, the component has just a transient internal 
state and/or stores persistent data via an external database component. 

Farm A task farm processes a stream of tasks {xq, . ■., Xm} producing a 
stream of results {/(xq), ..., f{xm)}- The computation of f{xi) is inde¬ 
pendent of the computation of f{xj) for any i ^ j (the task farm parallel 
pattern is often referred to as the "embarrassingly parallel" pattern). The 
items of the input stream are available at different times, in general: item 
Xi is available t > 0 time units after item Xi-i was available. Also, in 
the general case, it is not required that the output stream keeps the same 
ordering as the input stream, i.e. item f{xi) may be placed in the output 
stream in position j ^ i. In this case, in our farm Behavioural Skeleton, 
a stream of tasks is absorbed by a unicast S. Then each task is computed 
by one instance of W and the result is sent to C, which collects results 
according to afrom-any policy. This skeleton can be equipped with a self- 
optimizing policy as the number of W can be d 5 mamically changed in 
a sound way since they are stateless. The typical QoS goal is to keep a 
given limit (possibly d 5 mamically changing) of served requests in a time 
frame. Therefore, the AM just checks the average time tasks need to tra¬ 
verse the skeleton, and possibly reacts by creating/destroying instances 
of W, and wiring/unwiring them to/from the interfaces. 

Data-Parallel the task farm Behavioural Skeleton can be conveniently 
and easily adapted to cover other common patterns of parallel computa¬ 
tion. For example, data parallel computations can be captured by simply 
modifying the behavior associated with the S and C interfaces. In a data 
parallel computation a stream of tasks is absorbed by a scatter S. Each of 
the tasks appearing is split into (possibly overlapping) partitions, which 
are distributed to replicas of W to be computed. The results computed 
by the W are gathered and assembled by C in a single item, which is even¬ 
tually delivered onto the output stream. As in the previous case, the 
number of W can be dynamically changed (between different requests) 
in a sound way since they are stateless. In addition to the previous case. 
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the skeleton can be equipped with a self-configuration goal, e.g. resource 
balancing and tuning (e.g. disk space, load, memory usage), that can be 
achieved by changing the partition-worker mapping in S (and C, accord¬ 
ingly). 


The task farm (and dafa parallel) Behavioural Skelefon jusf ouflined 
can be easily modified fo fhe case in which fhe S is an RPC inferface. In 
fhis case, fhe C inferface can be eifher an RPC inferface or missing. Also, 
fhe sfafeless funcfional replicafion idea can be exfended fo fhe sfafeful 
case by requiring irmer componenfs W fo expose suifable mefhods fo 
serialize, read and wrife fhe infernal sfafe. A suifable manipulation of 
fhe serialized sfafe enables fhe reconfigurafion of workers (also in fhe 
dafa-parallel scenario (IT9l l. 


An 5 rway, in order fo achieve self-healing goals some additional re- 
quiremenfs on fhe GCM implemenfafion level should be enforced. They 
are relafed fo fhe implemenfafion of GCM mechanisms, such as compo- 
nenf membranes and fheir parfs (e.g. inferfaces) and messaging sysfem. 
Af fhe level of inferesf, fhey are primitive mechanisms, in which correcf- 
ness and robusfness should be enforced ex-anfe, af leasf fo achieve some 
of fhe described managemenf policies. 


The process of identification of ofher skelefons may benefif from fhe 
work done wifhin fhe software engineering communify which identified 
some common adapfation paradigms, such as proxies (l99t , which may be 
inferposed befween inferacfing componenfs fo change fheir inferacfion 
relafionships; and d 5 mamic wrappers (llllb . Bofh of fhese can be used for 
self-profecfion purposes. As an example, a couple of encrypting proxies 
can be used fo secure a communication befween componenfs. Wrapping 
can be used fo hide one or more inferfaces whefher a componenf is de¬ 
ployed info an unfrusfed plafform. 
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5.6 Autonomic Components: 
design and implementation 

The two main characteristics of autonomic components are the ability 
to self-manage and to cooperate with other autonomic components to 
achieve a common goal, such as guaranteeing a given behavior of an en- 
fire componenf-based application. In fhe lighf of fhis, viewing fhe man- 
agemenf of a single componenf as an atomic feafure enables design of 
ifs managemenf (fo a cerfain exfenf) in isolation. The managemenf of a 
single componenf is fherefore considered a logically centralized acfivify. 
Componenfs will be able fo inferacf wifh ofher componenfs according 
fo well-defined profocols described by managemenf interaction patterns, 
which are esfablished by fhe componenf model. 

5.6.1 The management of a GCM component 

The management of a single component is characterized by its ability to 
make non-trivial decisions. Thus GCM components are differentiated as 
being passive or active, with the following meanings: 

Passive A component exposes non-functional operations enabling intro¬ 
spection (state and sensors) and d 5 mamic reconfiguration. These 
operations exhibit a parametric but deterministic behavior. The op¬ 
eration semantics is not underpirmed by a decision making process 
(i.e. does not implement any optimization strategy), but can only 
be constrained by specific pre-conditions that, when not satisfied, 
may nullify an operation request. All components should imple¬ 
ment at least a reflection mechanism that may be queried about the 
list and the t5q3e of exposed operations. 

Active A component exhibits self-managing behavior, that is a further 
set of autonomic capabilities built on top of passive level function¬ 
ality. The process incarnates the autonomic management process: 
monitor, analyze, plan, execute. The monitoring phase is supported 
by introspective operations, while the executing phase is supported 
by re-configuring operations described above. 


134 



In the architecture of GCM components, these two features are im¬ 
plemented within the Autonomic Behaviour Controller (ABC) and Au¬ 
tonomic Manager (AM), respectively. Since the management is a logi¬ 
cally centralized activity, a single copy of each of fhem can appear in a 
componenf. Notice fhaf, fhis does nof prevenf a parallel implemenfafion 
of fhem for differenf reasons, such as faulf-folerance or performance. A 
passive componenf implemenfs jusf fhe ABC, whereas an acfive compo¬ 
nenf implemenfs bofh fhe ABC and fhe AM. The following relationship 
holds 


Comp < : PassiveComp < : ActiveComp 

where <: is a subfyping relafion. This is described in fhe CCM specifi- 
cafion by increasing values of conformance levels (|52}. 

GCM Passive Autonomic Components The ABC and the AM repre¬ 
sent two successive levels of absfracfion of componenf managemenf. As 
mentioned above, fhe ABC implemenfs operafions for componenf re¬ 
configuration and monitoring. The design of fhese operafions is sfricfly 
relafed fo membrane sfrucfure and implemenfafion, and fherefore fhe 
choice of implementing fhe ABC as a confroller in fhe membrane was 
the more obvious and natural. Within the membrane, the ABC can ac¬ 
cess all the services exposed by sub-component controllers, such as that 
related to life cycle and binding, in order to implement correct recon¬ 
figuration protocols. In general, these protocols depend on component 
structure and behavior. However, in the case of Behavioural Skelefons 
they depend almost solely on the skeleton family and nof on the partic¬ 
ular skeleton. In this regard, the ABC effectively abstracts out manage¬ 
ment operations for Behavioural Skelefons. 

As we presented Behavioural Skeletons based on the idea of func¬ 
tional replication, we show the details of fhese skelefons. In fhis case, fhe 
reconfigurafion operafions require fhe addition/removal of workers as 
well as fhe funing of disfribufion/collection sfrafegies used fo disfribufe 
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Non-Functional 
server ports 



Functional 
client port 


Figure 26: GCM: membrane and content (CC is the content controller, LC 
the lifecycle controller and BC is the binding controller). 


and collect tasks and results to and from the workers. The worker ad¬ 
dition and/or removal operations can be used to change the parallelism 
degree of the component as well to remap workers on different process¬ 
ing elements and/or platforms. The distribution/collection tuning op¬ 
erations can be used to throttle and balance the resource usage of work¬ 
ers, such as CPU, memory and lO. The introspection operations involve 
querying component status with respect to one or more pre-defined QoS 
metrics. The component status is generally obtained as a harmonized 
measure involving component status and inner component status. 

In the following we describe in some detail the implementation of a 
reconfiguration and an introspection operation. 

add-worker (k) Semantics: Add k workers to a skeleton based on the 
functional replication. 

1. Stop. The ABC requires the Lifecycle Controller (LC) to stop all the 
components. To this end, the LC retrieves from the Content Con¬ 
troller (CC) the list of inner components Wi • • • W„, and then issues 
a stop on them. 

2. Type Inspection. All the Wi • • • W„ have the same t 5 q)e. The ABC 
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retrieves from the CC the list of inner components Wi • • • W„, then 
retrieves T 5 ^eOf(Wi). 

3. New. One or more new inner components of t 5 rpe TypeOf(Wi) are 
created. 

4. Bind. The component server interface S is wired to newly created 
W„_|_i • • • W„+fc inner components via the Binding Controller (BC). 
W„_|_i • • • W„+fe, in turn, wire their client interfaces to the compo¬ 
nent collective client interface C. The process requires the inspec¬ 
tion of the t 5 q)es of the interfaces of Wi that is used again as a tem¬ 
plate for all W^. 

5. Restart. The ABC requires the LC to re-start all the components. 

6. Return. Return a failure code if some of the previous operations 
failed (e.g. inner components do not implement stop/start opera¬ 
tions); return success otherwise. 

get-measure (m) Semantics: Query the component about the current 
status of the measure m, which may depend on the status of the inner 
components (possibly involving other measures) and the membrane sta¬ 
tus. 

Examples: Transactions per unit time, load balancing, number of up-and- 
rurming workers, etc. 

1. Collect Workers' Measures. The ABC retrieves from the CC the list of 
inner components Wi • • • W„, then issues a getuneasure (m) on 
each. 

2. Collect Membrane Measures. The ABC queries membrane sensors 
relating to the particular metric m. 

3. Harmonize Measures. Measures acquired from workers and from the 
membrane are harmonized by using a m-dependent function (e.g. 
average, maximum, etc.). 
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4. Return. Return a failure code if some of fhe previous operations 
failed (e.g. sensor nof implemenfed in inner componenfs); refurn 
monifor information ofherwise. 


GCM Active Autonomic components The operations implemented in 
the ABC can be arbitrarily complex; however, they do not involve any 
decision making process. In general, each of fhem implemenfs a proto¬ 
col fhaf is a simple lisf of actions. On fhe confrary fhe AM is expected to 
enforce a confracfually specified QoS. To fhis end fhe AM should decide 
if a reconfiguration is needed, and if so, which reconfigurafion plan can 
re-esfablish confracf validify dlSll . Furfhermore, as we shall see in Secfion 
|5.6.2| fhe AM should also determine if fhe confracf violation is due to fhe 
managed componenf or is fhe byproducf of ofher componenfs' malfunc- 
fion. The archifecfure of an active GCM componenf is shown in Figure 
|2Zl 


Non-Functional 
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toward outer 
components 


read inner 
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sensors 


reconfigure 

inner 

passive 

component 
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catch exceptions 
from inner 
active 
components 


Figure 27: Left) GCM active component architecture. Right) ABC and AM 
interaction. 


The AM accepts a QoS contract^, which is currently defined as pair 
{V,E), where to is a sef of variables representing fhe measures fhe AM 

^the notion of QoS contract is still the subject of further investigations and possible re¬ 
finements. The one discussed here is the bare minimum necessary to discuss AM behavior 
and implementation. 
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can evaluate (via the ABC), and i? is a mathematical expression over 
these variables that might include the min and max operator over a fi¬ 
nite domain. The set of V defermines fhe minimum sef of measures fhe 
AM should be able fo monifor fo accepf fhe confracf. The E encodes fhe 
consfrainfs and goal fhe AM is required fo pursue. This encoding can 
be realized in many differenf ways provided E can be evaluafed in finife 
fime and possibly quife efficienfly. 

Having accepfed a QoS confracf, fhe AM iferafively checks ifs valid- 
ify, and in fhe case fhaf if appears broken, evaluafes a number of pre¬ 
defined reconfigurafion plans. Each reconfiguration plan consisfs of a 
sequence of actions (fo be execufed via fhe ABC), and a QoS forecasf 
formula. This formula allows fhe value of a subsef of V affer fhe recon¬ 
figurafion fo be forecasf. The AM insfanfiafes in fum all reconfigurafion 
plans obfaining, for each plan, a sef of forecasf values. A plan is marked 
as valid if fhe sef of V updafed wifh forecasf values satisfies fhe QoS con¬ 
fracf. Among fhe valid plans, fhe AM heurisfically chooses fhe recon¬ 
figurafion plan fo be execufed. If no reconfigurafion plan is valid, an 
exception is raised. 

As is clear, fhe main difficulfy in fhe AM detinifion is fhe specificafion 
of a reconfigurafion plan. In fhe general case, fhe reconfigurafion plans, 
and especially fheir forecasf formula, are sfricfly relafed fo fhe behav¬ 
ior of a particular componenf. As discussed in Section 5.3 Behavioural 
Skeletons enable fhe definition of reusable reconfigurafion plans by caf- 
egorizing and resfricfing componenf behavior in families and skelefons. 


5.6.2 Cooperative management 

The ulfimafe goal of QoS managemenf is fo guaranfee programmer in- 
fenfions despite soffware and environmenfal insfabilifies and malfunc- 
fions. To fhis end, fhe managemenf of a whole system should be coordi- 
nafed fo achieve a common goal. In general, we envisage a componenf- 
based system as a graph, whose nodes are componenfs, and edges are re¬ 
lations among fhem, such as dafa dependency, managemenf, geographic 
localify efc. Differenf relafions can be kepf disfincf by a proper label- 
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ing of edges. Here we restrict the focus to two relations which are of 
particular interest for GCM: used Joy and the implemented Joy (see Section 
|5.3) . Since the GCM is a hierarchical model, the nesting relation natu¬ 
rally defines the implemented Joy relationship. In particular, the applica¬ 
tion structure along the nesting relation describes a tree whose nodes 
represent components (leaves are primitive components) and edges rep¬ 
resent their nesting. In this case, the management of a composite compo¬ 
nent C is cooperatively performed by the AMc of the component itself 
and the AMc, of the child components Ci,i = \..n. In the case where 
irmer components are passive, the cooperation is really one of control by 
the outer component: services exposed by the ABCc^ are called by the 
ABCc. 

Conceptually, non-functional properties modeling run-time behavior 
of the whole hierarchy can be S 5 mthesized in a bottom-up fashion: the be¬ 
havior of a composite component depends on the behavior of its nested 
components. Management actions and QoS contracts should be pro¬ 
jected along the tree in a top-down fashion: the users usually would like 
to declare a global goal they expect from an application. This matches the 
idea of submitting a contract at the root of tree. A fully autonomic sys¬ 
tem should automatically split the global goal into sub-goals that should 
then be forced on irmer components. 

On the whole, each GCM component enforces local decisions. When 
a contract violation is detected, its AM tries autonomously to re-establish 
the contract to a valid status by re-configuring its membrane or irmer 
components. In the event that it carmot (no valid plan), it raises an event 
to its father component, thus increasing the extent of the reconfiguration. 
The overall behavior enforces the maximum locality of reconfigurations, 
which is a highly desirable property in a distributed system, since it eases 
the mapping of components onto the network of platforms that usually 
exhibit a hierarchical nature in terms of uniformity of resources and la¬ 
tency/bandwidth of networks (cluster of clusters). 

Observe that cooperation between components is unavoidable even 
in very simplistic applications. Let us consider an example: 
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Figure 28: Producer-filter-consumer with parallel filter (farm skeleton). 


Producer-filter-consumer Let us assume that the application sketched 
in Figure has the final goal to generate, render, and display a video 
with a given minimum number of frames/sec {FPS > k). The con¬ 
tract is split into three identical contracts since the property should be 
enforced on all stages in order to hold globally. The rendering (filter) 
has been parallelized since it is the most CPU-demanding stage. Two 
common problems of such applications are a transient overload of plat¬ 
form where Wi • • • W„ are running, or an increased complexity of scene 
to be rendered. These events may lead to a violation of QoS contract at 
the AMf- In this case, it may increase the number of workers (mapped 
on fresh machines) to deal with the insufficient aggregate power of al¬ 
ready running resources. In many cases this will locally solve the prob¬ 
lem. However, a slightly more sophisticated contract should consider 
also the input and output channels. In particular the filter stage might be 
not rendering enough frames because it does not receive enough scenes 
to render. In this case the AMf can detect the local violation, but can¬ 
not locally solve the problem. As a matter of fact, no plan involving a 
change of parallelism degree can solve this problem. AMf can just sig¬ 
nal the problem to a higher level AM, 4 , which can try to remap the input 
channel to a faster link, or simply signal to the end user that the contract 
is not satisfied. 
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Figure 29: Reconfiguration overhead: Stop. 


5.7 Experiments 

In order to validate the Behavioural Skeletons approach, we conducted 
some experiments with the current protot 5 rpe of the GCM. It is under de¬ 
velopment in the GridCOMP STREP project (|3). The protot 5 rpe, which is 
being developed on top of ProAcfive middleware dlOSII , includes almosf 
all of fhe feafures described in fhis chapfer. All fhe experimenfal dafa 
are measured on fhe applicafion shown in Figure|^fhaf we already pre¬ 
sented in the previous section. It basically is a three-stages pipeline in 
which the second stage consists in a farm of workers processing fhe im¬ 
ages coming from the first stage, and delivering them to the third stage. 
The experiments mainly aim to assess the overhead due to management 
and reconfiguration of GCM componenfs. For fhe sake of reproducibilify, 
the experiments have been run on a cluster instead of a more heferoge- 
neous grid. The clusfer includes 31 nodes (1 Infel P3@800MHz core per 
node) wired wifh a fasf Efhemef. Workers are allocafed in fhe clusfer in 
a round robin fashion wifh up fo 3 workers per node (for a fofal of 93 
workers). Nofe however, fhe very same experimenfal code can run on 
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Figure 31: Reconfiguration overhead: Restart. 
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any distributed platform supported by the ProActive middleware. 

Figures and respectively show the time spent on the farm 

Behavioural Skeleton (filler) for fhe stop, new and restart Aufonomic Be¬ 
havioural Confroller (ABC) services described in Secfion [5.6.1| This lime 
consisfs in applicafion overhead, since in current implementation none 
of fhe workers can accepf new fasks during fhe process. In fhe figures, a 
point k in the X-axis describes the overhead due to stop/new/restart in the 
adaptation of fhe running program from a A: to fc -|- 1 worker configura¬ 
tion. As highlighted by the curves in Figure and [ST] the overhead of 
stop and restart is linear wifh respecf to fhe number of workers involved 
in fhe operafions. This is mainly due to a linear fime barrier wifhin fhe 
Life cycle Confroller (LCC), which is an inherenf parf of fhe underlying 
ProAcfive middleware. Indeed, in fhe currenf implemenfafion fhe LCC 
sequentially slops all fhe workers. Nofe fhaf adapfafion process does nof 
sfricfly require such a barrier. Bofh slopping all fhe workers and linear 
fime S 5 mchronizafion are peculiarifies of fhe currenf GCM implemenfa¬ 
fion on fop of fhe ProAcfive middleware, and nof of fhe farm Behavioural 
Skeleton, which can be implemenfed avoiding bofh problems. In addi¬ 
tion, fhe creation of a new worker can be executed, af teas! in principle, 
oufside fhe critical pafh by using a speculafive creafion. 


Figure shows fhe fime spenf for the new Autonomic Behavioural 
Controller (ABC) operation (see Section 5.6. 1[ . Again, in this case, the 
time is overhead. The experiment measures the time required for fhe 
creafion of a single worker, and fhus fhe times measured are almosf in- 
dependenf of fhe number of workers pre-existing fhe new one. 


As highlighfed by fhe Figure [30| and [31] fhe overhead of fhe new and 
restart operafions is much higher in fhe case where a fresh plafform is 
involved (number of workers less fhan 32). The difference is mainly 
due fo fhe additional fime for Java remote class loading. In facf, when 
a worker is creafed, if fhe classes if needs are nof presenf (in fhe machine 
thaf is running if), fhey are copied locally fhen loaded in fhe clusfer node 
main memory and compiled. Clearly, performing such operations re¬ 
quire time, hundreds of milliseconds. Rafher, if fhe classes are already 
presenf, already loaded in main memory or even already compiled in 
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Figure 32: Self-optimization experiment. 


machine target code by the Java JIT, performing these reconfiguration 
operations is noticeably cheaper. 

The results of the last experiment are presented in Figure It de¬ 
scribes the behavior of the application over quite a long run (two hours, 
approximately) that includes several self-triggered reconfigurations. In 
this case the application is provided with a Quality of Service (QoS) con¬ 
tract that enforces the production of a minimum of 1.5 results per second 
(tasks/s). During the run, an increasing number of platforms are exter¬ 
nally overloaded with an artificial load (we started the compilation of 
some complex software written in C++). The top half of the figure re¬ 
ports the measured average throughput of the filter stage (the second, 
actually), and the QoS contract. The bottom half of the figure reports 
the number of overloaded machines along the run, and the correspond¬ 
ing increase of workers of the filter stage. Initially the throughput of the 
filter stage is abundantly higher than requested (~ 3.5 tasks/s); but it 
decreases when more machines are overloaded. As soon as the contract 
is violated, the Autonomic Manager reacts by adding more workers. 
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Summarizing the Chapter 


The challenge of autonomicity in the context of component-based develop¬ 
ment of grid software is substantial. Building into components autonomic ca¬ 
pability typically impairs their reusability. In this Chapter we proposed Be¬ 
havioural Skeletons as a compromise: being skeletons they support reuse, while 
their parameterization allows the controlled adaptivity needed to achieve dy¬ 
namic adjustment of QoS while preserving functionality. We also presented 
a significant set of skeletons and we discussed how Behavioural Skeletons can 
be implemented in the framework of the GCM component model. Behavioural 
Skeletons provide the programmer with the ability to implement autonomic man¬ 
agers completely taking care of the parallelism exploitation details by simply in¬ 
stantiating existing skeletons and by providing suitable, functional parameters. 
Finally, we discussed the experimental results achieved when running an appli¬ 
cation exploiting instances of our Behavioural Skeletons and we showed how the 
skeletons used may take decisions at the appropriate time to maintain the appli¬ 
cation behavior within the limits stated by the user with a specific performance 
contract. The whole experiments have been performed using GCM components 
and Behavioural Skeletons, as being designed and implemented in the framework 
of the CoreGRID and GridCOMP projects. To our knowledge, no other similar 
results are available yet. 
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Chapter 6 

Conclusions 


Over the years, a lot of models and tools for parallel programming have 
been proposed. This greaf deal of efforfs is mainly due fo fhe difficul- 
fies in coordinafing several, possibly hundreds or fhousands, activities 
in an easy way buf allowing an efficienf exploifafion of compufafional 
resources. In facf, fo dafe does nof exisf a universal approach working 
better fhan ofhers in every sifuafion. Acfually, fhere are several good 
approaches based on differenf perspectives and absfracfion levels. Nev- 
erfheless, sfarfing from fhe second half of ninefies, wifh fhe advenf of 
compufafional Grids, parallel programming difficulfies became greafer 
and greafer and also fhe mosf promising approaches frail along. Indeed, 
programming fhe Grids is even more difficulf fhan fradifional parallel 
programming. This is because fhe compufers belonging fo a Grid can be 
heferogeneous, separafed by firewalls, unsafe and managed by differenf 
adminisfrafion policies. To address fhese addifional difficulfies mosf of 
fhe models and fools conceived and developed for parallel programming 
have fo be re-fhoughf and adapfed. In particular, Sfrucfured Parallel Pro¬ 
gramming models, and fhe derived environmenf have been proved fo be 
very effecfive approach for programming parallel applications, buf some 
well-known issues prevenf fhem from achieving significanf popularify in 
fhe wider parallel and grid programming communify. 

In fhis fhesis we presenfed an organic sef of fools and models con- 
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ceived, designed and developed or properly modified to address most 
of these issues. 

We started discussing how we modified the muskel framework for 
supporting the issue related to the lack of extendability of the skeleton 
systems. We discussed how our customized muskel supports the in¬ 
troduction of new skeletons, modeling parallelism exploitation patterns 
not originally covered by the primitive muskel skeletons. This possibil¬ 
ity is supported by allowing muskel users (the programmers) to define 
new skeletons providing the arbitrary data flow graph executed in the 
skeleton and by letting our muskel version to seamlessly integrate such 
new skeletons with the primitive ones. We also presented experimen¬ 
tal results validating our muskel approach to extend and customize its 
skeleton set. We ran several test programs using the custom features 
introduced in muskel. When grain is small, muskel does not scale 
well, even using a very small number of remote interpreter instances. 
When the computational grain is high enough the efficiency is definitely 
close to the ideal one. Despite the data shown in this thesis refer to 
S 5 mthetic computations, the tests we conducted using actual computa¬ 
tions achieved very similar results. This because the automatic load bal¬ 
ancing mechanism implemented in the muskel distributed interpreter 
through auto scheduling perfectly optimized the execution of variable 
grain macro data-flow instructions. As far as we know, this is the most 
significant effort in the skeleton community to tackle problems deriving 
from a fixed skeleton set. Only Schaeffer and his group at the University 
of Alberta implemented a system were programmers can, in controlled 
ways, insert new parallelism exploitation patterns in the system (l38t , al¬ 
though the approach followed here is a bit different, in that programmers 
are encouraged to intervene directly in the run-time support implemen¬ 
tation, to introduce new skeletons, while in muskel new skeletons may 
be introduced using the intermediate macro data flow language as the 
skeleton "assembly" language. Unfortunately, programmers using this 
approach, in order to program unstructured parallel application, have 
to interact directly with data-flow graph. It requires to programmers to 
reason in terms of program-blocks instead of a monolithic program. In 
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order to ease the generation of macro data-flow blocks and in general fo 
provide mechanism easing fhe use of sfrucfured parallel programming 
environmenf, we exploifed some metaprogramming fechniques. 

We exploifed some mefaprogramming fechniques based bofh on As- 
pecf Orienfed Programming (AOP) and on Affribufe Orienfed Program¬ 
ming (@OP). We showed how fhese fechniques can be seamlessly ex¬ 
ploifed fo fransform sequenfial applicafions info parallel ones. In parfic- 
ular, we showed how annofafions and aspecf can be exploifed fo drive 
fhe sequenfial applicafion fransformafion info a macro dafa-flow graph 
fhaf can be execufed on disfribufed archifecfures. The exploifafion of 
@OP and AOP fechniques allows fo complefely separafe fhe concerns 
relafive fo parallelism exploifafion and applicafion funcfional code. In 
parficular, fhe same applicafion code used fo perform funcfional debug¬ 
ging on a single, sequenfial machine may be easily fumed info parallel 
code. To validafe fhe @OP approach we implemenfed PAL, a java an- 
nofafion based mefaprogramming framework fhaf resfrucfures applica¬ 
fions af byfecode-level af run-fime in order fo make fhem parallel. PAL 
fransformafions depend on: i) fhe resources available af run-fime, ii) 
fhe hinfs provided by programmers and iii) fhe available adapters. An 
adapfer is a specialized enfify fhaf insfrucfs fhe PAL fransformafion en¬ 
gine fo drive fhe code fransformafion depending on fhe available par¬ 
allel fools and frameworks. Experimenfal resulfs show fhaf fhe PAL is 
an effecfive and efficienf approach for handling resource heferogeneify 
and d 5 mamicify. Acfually run-fime code fransformafion brings fo a very 
good exploifafion of compufafional resources. For fhis implemenfafion 
we developed two disfincf adapfers. The firsf adapfer we developed 
fosfer fhe byfecode fransformafion making fhe original code a mulfi- 
fhreaded one. The ofher adapfer supporfs fhe byfecode fransformafion 
fhaf makes fhe original code complianf wifh JJPF, a sfrucfured parallel 
programming framework we developed some years ago. PAL demon- 
sfrafed fhaf, given fhe exisfence of a proper mefaprogramming run-fime 
supporf, armofafions are a handy way bofh fo indicafe which parfs of 
a program musf run in parallel and fo express non-funcfional require- 
menfs direcfly in fhe source code. Therefore, we decided fo apply fhe 
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main features of PAL approach to our modified muskel implementa¬ 
tion. Actually, adapting them to muskel we changed a little bit the 
approach. Such a change is due to a few motivations. First of all be¬ 
cause muskel provides per se a distributed macro data-flow executor 
whereas PAL exploits external tools for distributed program execution. 
Moreover, we would like to have a more flexible mechanism for macro 
data-flow block generation and management. Finally, we would like to 
exploit a standard tool for run-time code transformation instead of using 
ad-hoc tools, like the one we developed for PAL. As a consequence we 
decided to use integrate in muskel the AOP model and in particular the 
AspectJ framework. The integration has been performed in two steps, in 
the first step we integrated the AOP mechanisms in order to achieve very 
simple code transformation. The second step consisted in testing the in¬ 
tegration of muskel with AspectJ to in a more complex scenario. Hence, 
we exploited the aspect oriented programming support we integrated in 
muskel in order to develop workflows which structure and processing 
are optimized at run-time. In order to prove the effectiveness of the ap¬ 
proach in muskel, we conducted some experiments on a network of 
workstations. The only difference between plain muskel and the sys¬ 
tem proposed here to execute workflows lies in the way fireable instruc¬ 
tions are provided to the distributed data-flow interpreter of muskel. 
Indeed, in plain muskel, fireable instructions are taken from a compiled 
representation of a data-flow graph. Each time a new token arrives to 
a macro data-flow instruction in the graph the target data-flow instruc¬ 
tion is checked for "fireability" and, possibly, delivered to the distributed 
macro data-flow interpreter. The time spent is in the sub-micro second 
range (net time, not taking into account time spent to copy parameters 
in memory during the interpreter call). When executing workflows ac¬ 
cording to the approach discussed here, instead, fireable instructions is 
generated at run-time by the AOP engine. We measured the overhead 
when exploiting the AOP approach, it is approximately 23 milliseconds 
per workflow node. 

These two results presented are feasible approaches for programming 
cluster or networks of workstation but are not suitable for computational 
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Grids, where component models are preferable. This is due to several 
motivations we described in deep in this thesis. Provide parallel pro¬ 
gramming models for Grids are important because they are becoming 
the dominant t 5 rpe of parallel architectures. Moreover, due to their het¬ 
erogeneous and distributed nature, they represent a very good test-bed 
for testing parallel programming models dealing with dynamicity han¬ 
dling. The muskel framework, handle dynamicity exploiting the Ap¬ 
plication Manager: an entity that observes the behavior of the parallel 
application and in case of problems reacts aiming to fix them. This ap¬ 
proach has proved to be effective. Nevertheless, some of the implemen¬ 
tation choices done when muskel was developed limit its exploitation 
on Grids. Therefore, we decided to generalize and extend the muskel 
Application Manager approach to make it suitable for components mod¬ 
els, in order to be able to port the approach in existing component mod¬ 
els. We ported the muskel approach in the Grid Component Model. 
Actually, the Application Manager approach form the base of the auto¬ 
nomic features of GCM: each self-optimizing GCM component contains 
an Application Manager that in GCM is called Autonomic Manager. Nev¬ 
ertheless, Autonomic Manager rely fully on the application programmer's 
expertise for the setup of the management code, which can be quite diffi¬ 
cult to write since it is tailored for the particular component or assembly 
of them. As a result, the introduction of dynamic adaptivity might enable 
the management of grid d 5 mamism but, at the same time, decreases the 
component reuse potential since it further specializes components with 
application specific management code. In order to address this prob¬ 
lem, we proposed the Behavioural Skeletons as a novel way to describe 
autonomic components in the GCM framework. Behavioural Skeletons 
aim to describe recurring patterns of component assemblies that can be 
equipped with correct and effective management strategies with respect 
to a given management goal. The Behavioural Skeletons model provides 
a way for handling d 5 mamicity, supporting reuse both of functional and 
non-functional code. We presented a significant set of skeletons and we 
discussed how behavioural skeletons can be implemented in the frame¬ 
work of the GCM component model. Behavioural skeletons provide the 
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programmer with the ability to implement autonomic managers com¬ 
pletely taking care of the parallelism exploitation details by simply in¬ 
stantiating existing skeletons and by providing suitable, functional pa¬ 
rameters. To validate our Behavioural Skeletons we conducted some ex¬ 
periments with the current prototype of fhe GCM fhaf is currenfly un¬ 
der developmenf in fhe GridCOMP STREP projecf (|3]|. We discussed fhe 
experimenfal resulfs achieved when running an applicafion exploifing 
insfances of our Behavioural Skelefons and we showed how fhe skele¬ 
tons used may take decisions at the appropriate time to maintain the 
application behaviour within the limits stated by the user with a specific 
performance contracf. 


Future Works 

New efforfs for fufure work can be invesfed in differenf directions, as 
suggesfed by fhe resulfs offered by fhis fhesis. 

Concerning fhe macro dafa-flow based skeleton cusfomizations, new 
mechanisms for modifying fhe macro dafa-flow graph can be conceived, 
possibly simpler fhan fhe existing one. Jusf as a nofe, currenfly we are 
developing a graphic fool fhaf allows programmers ( muskel users) to 
design fheir macro dafa-flow graphs and fhen compile fhem direcfly to 
Java code as required by muskel. 

Several ofher annofafions and aspecfs can be designed and imple¬ 
mented for easing fhe run-fime generafion of macro dafa-flow blocks. 
Possibly supporting several fypes of non-funcfional requiremenfs. Re¬ 
garding PAL, many adapfers, even more complex fhan exisfing one can 
be developed. In particular, adapfers for widely-used frameworks for 
Grid programming, like Globus or ProAcfive. Anofher inferesfing possi- 
bilify can be fhe porfing of fhe adapfers model in our customized muskel, 
perhaps making possible fhe fransformafion, af run-fime, of fhe macro 
dafa-flow blocks generated by muskel in GCM componenfs. 

In fhis fhesis we presenfed a reduced sef of Behavioural Skeletons, 
ofher skelefons can be conceived, designed and implemenfed. As an ex¬ 
ample, a Behavioural Skelefon supporting fhe non-funcfional replication 
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management for easing the development of fault-tolerant component ap¬ 
plications. Furthermore, a lot of research can be conducted on the dis¬ 
tributed (cooperative) self-management of component applications, in 
particular regarding to the methodologies for splitting the user specified 
QoS contracts. 


153 



154 



References 


[1] Ada programming language specification, http://www.open- 
std.org/jtcl/sc22/wg9/. 1^ 


[2] Grid5000projectwebpage.www.grid5000.fr/. 63 


[3] Gridcomp eu strep project website, http: / /gridcomp.ercim.org. 
[631 [1421 [ikl 


[4] Jini website, http://www.jini.org. 54 


[5] Common component architecture forum home page, 
^.acl.lanl.gov/cca/, 2003. |6 63 


WWW.i 


[6] Aspectj home page, 2007. http://www.eclipse.org/aspectj/. 62 


[7] ECMA 335. Common languageinfrastructure(cli). 

http: / / www.ecma.ch/ecmal / STAND / ecma-335.htm, 2001. 


[8] Java specification requests 175: A metadata facility for the java pro¬ 


gramming language. http://www.jcp.org, September 2004. 61 94 

[9] Harold Abelson and Gerald J. Sussman. Structure and Interpretation 


of Computer Programs. MIT Press, Cambridge, MA, USA, 1996. 30 


[10] W.B. Ackerman. Dataflow languages. Computer, 15(2):50-69,1982. 


155 













[11] M. Aldinucci and M. Danelutto. An operational semantics for 
skeletons. In G.R. Joubert, W.E. Nagel, F.J. Peters, and W.V. Wal¬ 
ter, editors. Parallel Computing: Software Technology, Algorithms, Ar¬ 
chitect ures and Applications, Advances in Parallel Computing. Else¬ 
vier, The Netherland, 2004. 


[12] M. Aldinucci, M. Danelutto, and P. Teti. An advanced environment 
supporting structured parallel programming in Java. Future Gen¬ 


eration Computer Systems, 19(5):611-626, 2003. Elsevier. 48 50 53 


[13] Marco Aldinucci, Erancoise Andre, Jeremy Buisson, Sonia Campa, 
Massimo Coppola, Marco Danelutto, and Corrado Zoccolo. Paral¬ 
lel program/component adaptivity management. In C. R. Joubert, 
W. E. Nagel, E. J. Peters, O. Plata, P. Tirado, and E. Zapata, editors. 
Parallel Computing: Current & Future Issues of High-End Computing, 
Proc. of PARCO 2005, volume 33 of NIC, pages 89-96, Germany, 
December 2005. Research Cenfre Jiilich. |138| 

[14] Marco Aldinucci, Sonia Campa, Pierpaolo Ciullo, Massimo Cop¬ 
pola, Marco Danelutto, Paolo Pesciullesi, Roberto Ravazzolo, Mas¬ 
simo Torquafi, Marco Varmeschi, and Corrado Zoccolo. A frame¬ 
work for experimenfing wifh sfrucfure parallel programming en- 
vironmenf design. In G. R. Jouberf, W. E. Nagel, E. J. Peters, and 
W. V. Waller, edifors. Parallel Computing: Software Technology, Al¬ 
gorithms, Architectures and Applications, PARCO 2003, volume 13 of 
Advances in Parallel Computing, pages 617-624, Dresden, Germany, 
2004. Elsevier. 

[15] Marco Aldinucci, Sonia Campa, Pierpaolo Ciullo, Massimo Cop¬ 
pola, Silvia Magini, Paolo Pesciullesi, Laura Pofifi, Roberfo Ravaz¬ 
zolo, Massimo Torquafi, Marco Varmeschi, and Corrado Zoccolo. 
The implemenfafion of ASSIST, an environmenf for parallel and 
disfribufed programming. In H. Kosch, L. Boszormenyi, and 
H. Hellwagner, edifors. Proceedings of the 9th International Euro-Par 


156 







Conference, volume 2790 of Lecture Notes in Computer Science, pages 


712-721, Klagenfurt, Austria, August 2003. Springer Verlag. 70 


[16] Marco Aldinucci, Sonia Campa, Marco Danelutto, Patrizio Dazzi, 
Peter Kilpatrick, Domenico Laforenza, and Nicola Tonellotto. Be¬ 
havioural skeletons for component autonomic management on 
grids. In Marco Danelutto, Paraskevi Frangopoulou, and Vladimir 
Getov, editors. Making Grids Work, CoreGRID. Springer Verlag, 
June 2008. |7l[T^ 


[17] Marco Aldinucci, Sonia Campa, Marco Danelutto, Marco Van- 
neschi, Peter Kilpatrick, Patrizio Dazzi, Domenico Laforenza, and 
Nicola Tonellotto. Behavioral skeletons in gem: automatic manage¬ 
ment of grid components. In Julien Bourgeois and Didier El Baz, 
editors. Proceedings of the 16th International Euromicro Conference on 
Parallel, Distributed and Network-based Processing, pages 54-63. IEEE 


Computer Society Press, Toulose, Prance, Pebruary 2008. 3 7 125 


[18] Marco Aldinucci and Marco Danelutto. Stream parallel skeleton 
optimization. In Proceedings of the PDCS: International Conference 
on Parallel and Distributed Computing and Systems, pages 955-962, 
Cambridge, Massachusetts, USA, November 1999. lASTED, ACTA 


press. 51 62 76 


[19] Marco Aldinucci and Marco Danelutto. Algorithmic skeletons 
meeting grids. Parallel Computing, 32(7):449-462, 2006. 
|57l|60l [T^[T^[T33| 


[20] Marco Aldinucci and Marco Danelutto. Skeleton based parallel 
programming: functional and parallel semantic in a single shot. 
Computer Languages, Systems and Structures, 2006. 


[21] Marco Aldinucci, Marco Danelutto, and Patrizio Dazzi. Muskel: 
an expandable skeleton environment. Scalable Computing: Practice 
and Experience, 8(4):325-341, December 2007. |3 


[22] Marco Aldinucci, Marco Danelutto, and Marco Vanneschi. Auto¬ 
nomic QoS in ASSIST grid-aware components. In Beniamino Di 


157 





















Martino and Salvatore Venticinque, editors. Proceedings of the 
14th International Euromicro Conference on Parallel, Distributed and 
Network-based Processing, pages 221-230, Montbeliard, France, 
February 2006. IEEE. 124| 


[23] E. Andre, J. Buisson, and J.-L. Pazat. Dynamic adaptation of 
parallel codes: toward self-adaptable components for fhe Grid. 
In V. Gefov and T. Kielmann, editors. Proceedings of the Interna¬ 
tional Workshop on Component Models and Systems for Grid Applica¬ 
tions, GoreGRID series, IGS '04, Sainf-Malo, Erance, January 2005. 


Springer Verlag. 124 


[24] A. Andrzejak, A. Reinefeld, E. Schintke, and T. Schiiff. On adapf- 
abilify in grid sysfems. In V. Gefov, D. Laforenza, and A. Reinefeld, 
edifors. Future Generation Grids, GoreGRID series. Springer Verlag, 
November 2005. [US 


[25] Rob Armsfrong, Dennis Gannon, A1 Geisf, Kafarz 5 ma Keahey, 
Scoff R. Kohn, Lois Mclnnes, Sfeve R. Parker, and Brenf A. 
Smolinski. Toward a common componenf archifecfure for high- 
performance scienfific compufing. In HPDC, 1999. 


[26] Arvind and D.E Guller. Dafaflow archifecfures. Technical reporf, 
Armual Reviews of Gompufer Science, 1986. 


27 


[27] Arvind and K. Ekanadham. Eufure scienfific programming on 
parallel machines. Journal of Parallel and Distributed Computing, 
5(5):460M93,1988. [^ 


[28] Krsfe Asanovic, Ras Bodik, Bryan Ghrisfopher Cafanzaro, 
Joseph James Gebis, Parry Husbands, Kurf Keufzer, David A. Paf- 
ferson, William Lesfer Plishker, John Shall, Samuel Webb Williams, 
and Kafherine A. Yelick. The landscape of parallel compufing re¬ 
search: A view from berkeley. Technical Reporf UCB/EEGS-2006- 
183, EECS Deparfmenf, Universify of Galifornia, Berkeley, Dec 

2006 . ns 


158 








[29] P. Au, J. Darlington, M. Ghanem, Y. Guo, H.W. To, and J. Yang. 
Go-ordinating heterogeneous parallel computation. In L. Bouge, 
P. Fraigniaud, A. Mignotte, and Y. Robert, editors, Europar '96, 


pages 601-614. Springer Verlag, 1996. 48 


[30] Bruno Bacci, Marco Danelutto, Salvatore Orlando, Susanna Pela- 
gatti, and Marco Vanneschi. P^L: a structured high level program¬ 
ming language and its structured support. Concurrency Practice and 


Experience, 7(3):225-255, May 1995. 48 


[31] Bruno Bacci, Marco Danelutto, Salvatore Orlando, Susanna Pela- 
gatti, and Marco Vanneschi. P31: A structured high level program¬ 
ming language and its structured support. Concurrency: Practice 


and Experience, 7(3):225-255, May 1995. 49 


[32] Bruno Bacci, Marco Danelutto, Susanna Pelagatti, and Marco Van¬ 
neschi. SklE: A heterogeneous environment for HPG applications. 


Parallel Computing, 25(13-14):1827-1852,1999. 48 66 


[33] P. V. Bangalore. Generating parallel applications for disfribufed 
memory sysfems using aspecfs, componenfs, and pafferns. In The 
6th AOSD Workshop on Aspects, Components and Patterns for Infras¬ 
tructure Software (ACP4IS), Vancouver, BC, Canada, March 2006. 
ACM. EH 


[34] J. Barklund. Parallel Unification. PhD fhesis, Uppsala Universify, 
1989. m 


[35] Anne Benoif, Murray Cole, Sfephen Gilmore, and Jane Hillsfon. 
Flexible skelefal programming wifh eskel. In Euro-Par, pages 761- 
770, 2005. 

[36] Daniel G. Bobrow, Richard P. Gabriel, and Jon L. While. Clos in 
confexf: The shape of fhe design space. In Andreas Paepcke, edifor. 
Object-oriented Programming, pages 29-61. MIT Press, Cambridge, 
MA, USA, 1993. ED 


159 








[37] Jan Bosch. Superimposition: a component adaptation technique. 
Information and Software Technology, 41(5):257-273,1999. 124| 


[38] S. Bromling. Generalising pattern-based parallel programming 
systems. In G. R. Joubert, A. Murli, F. J. Peters, and M. Varmeschi, 
editors. Parallel Computing: Advances and Current Issues. Proceed¬ 
ings of the International Conference ParCo2001, pages 91-100. Impe¬ 
rial College Press, 2002. [58lpl[88l[T48l 


[39] E. Bruneton, T. Coupaye, and J-B. Stefani. The Fractal Component 
Model, Technical Specification. ObjectWeb Consortium, 2003. 


125 


[40] Coupaye T Bruneton E, Lenglet R. Asm: a code manipulation tool 
to implement adaptable systems, grenoble, trance. Adaptable and 
Extensible Component Systems, Nov. 2002. 102| 


[41] Bill Burke and Richard Monson-Haefel. Enterprise JavaBeans 3.0. 


O'Reilly, 5th edition, 2006. 61 


[42] D. Caromel. Service, as 5 mchrony, and wait-by-necessity. Journal of 


Object-Oriented Programming, Nov/Dec 1989. 103 


[43] D. Caromel, L. Henrio, and B. Serpette. As 5 mchronous and deter¬ 


ministic objects, 2004. 100 


[44] Denis Caromel and Ludovic Henrio. A Theory of Distributed Object. 


Springer Verlag, 2005. 100 


[45] Walter Cazzola, Antonio Cisternino, and Diego Colombo. [a]c#: 
C# with a customizable code armotation mechanism. In SAC '05: 
Proceedings of the 2005 ACM symposium on Applied computing, pages 
1264-1268, New York, NY, USA, 2005. ACM. |95] 

[46] Paolo Ciancarini. Blackboard programming in shared prolog. In 
Selected papers of the second workshop on Languages and compilers for 
parallel computing, pages 170-185, London, UK, UK, 1990. Pitman 


Publishing. 26 


160 




















[47] S. Ciarpaglini, M. Danelutto, L. Folchi, C. Manconi, and S. Pela- 
gatti. ANACLETO: a template-based P3L compiler. In Proceedings 
of the Parallel Computing Workshop (PCW'97), 1997. Camberra, Aus¬ 
tralia. 


[48] Keith Clark and Steve Gregory. Parlog: parallel programming in 


logic. ACM Trans. Program. Lang. Syst., 8(l):l-49,1986. 26 


[49] M. Cole and A. Benoit. The eSkel home page, 2005. http:// 
homepages.inf.ed.ac.uk/abenoit1/eSkel/ m 

[50] Murray Cole. Algorithmic skeletons: structured management of parallel 

|g|4^|M 


computation. MIT Press, Cambridge, MA, USA, 1991. 


[51] Murray Cole. Bringing skeletons out of the closet: A pragmatic 
manifesto for skelefal parallel programming. Parallel Computing, 
30(3):389-406, 2004. |^|g[T4j|^|^|^|^|^ 


[52] CoreGRID NoE deliverable series, Insfifufe on Programming 
Model. Deliverable D.PM.04 - Basic Features of the Grid Component 


Model (assessed), Pebruary 2007. 6 63 124 125 126 135 


[53] G. A. Gunha and J. L. Sobral. An annofafion-based framework for 
parallel compufing. In Proceedings of the 15th International Euromi¬ 
cro Conference on Parallel, Distributed and Network-based Processing, 


pages 113-120, Los Alamifos, CA, USA, 2007. IEEE. 96 


[54] Krzyszfof Gzamecki and Ulrich W. Eisenecker. Generative Program¬ 
ming - Methods, Tools, and Applications. Addison-Wesley, June 2000. 

[Tool 


[55] M. Daneluffo, R. Di Gosmo, X. Leroy, and S. Pelagaffi. Parallel 
Puncfional Programming wifh Skelefons: fhe OGAMLP3L experi- 


menf. In ACM Sigplan Workshop on ML, pages 31-39,1998. 48 


[56] Marco Daneluffo. D 5 mamic run time support for skelefons. In E. H. 
D'Hollander, G. R. Jouberf, E. J. Refers, and H. J. Sips, edifors. Pro¬ 
ceedings of the International PARCO 99: Parallel Computing, Parallel 


161 































Computing Fundamentals & Applications, pages 460^67. Impe¬ 
rial College Press, 1999. |63|^|^ 

[57] Marco Danelutto. Efficient support for skelefons on worksfafion 


clusfers. Parallel Processing Letters, ll{iyAl-56,2001. 50 


[58] Marco Daneluffo. QoS in parallel programming fhrough applica¬ 
tion managers. In Proceedings of the 13th International Euromicro Con¬ 
ference on Parallel, Distributed and Network-based Processing, pages 


282-289, Lugano, Swifzerland, February 2005. IEEE. 50 76 77 118 


[59] Marco Daneluffo and Pafrizio Dazzi. A java/jini framework sup- 
porfmg sfream parallel compulations. In Proceedings of the Interna¬ 
tional PARCO 2005: Parallel Computing, volume 33 of John von Neu¬ 
mann Institute for Computing Series, pages 681-688. Cenfral Insfifufe 
for Applied Mafhemafics, Jiilich, Germany, Sepfember 2005. 


[60] Marco Daneluffo and Pafrizio Dazzi. Workflows on fop of a 
macro dafa flow inferprefer exploiting aspecfs. In Marco Dane¬ 
luffo, Paraskevi Frangopoulou, and Vladimir Gefov, editors. Mak¬ 
ing Grids Work, CoreGRID. Springer, June 2008. |3 

[61] Marco Daneluffo, Marcelo Pasin, Marco Vanneschi, Pafrizio Dazzi, 
Luigi Presfi, and Domenico Laforenza. Pal: Exploiting java an- 
nofafions for parallelism. In Marian Bubak, Sergei Gorlafch, and 
Thierry Priol, edifors. Achievements in European Research on Grid 
Systems, CoreGRID Series, pages 83-96. Springer, Krakow, Poland, 
2007 . 111 ^ 


96 


[62] Marco Daneluffo and Massimiliano Sfigliani. SKElib: parallel pro¬ 
gramming wifh skelefons in G. In A. Bode, T. Ludwing, W. Karl, 
and R. Wismiiller, edifors, Proc. of 6th Inti. Euro-Par 2000 Parallel 
Processing, volume 1900 of Lecture Notes in Computer Science, pages 


1175-1184, Munich, Germany, Augusf 2000. Springer Verlag. 48 


[63] Marco Daneluffo and P. Tefi. Lifhium: A sfrucfured parallel pro¬ 
gramming environmenf in java. In fees '02: Proceedings of the Inter- 


162 
















national Conference on Computational Science-Part 11, pages 844-853, 


London, UK, 2002. Springer-Verlag. 50 


[64] J. Darlington, A. J. Field, P.G. Harrison, P. H. J. Kelly, D. W. N. 
Sharp, R. L. While, and Q. Wu. Parallel programming using skele¬ 
ton functions. In Proc. of Parallel Architectures and Langauges Europe 
(PARLE'93), volume 694 of Lecture Notes in Computer Science, pages 


146-160, Munich, Germany, June 1993. Springer Verlag. 48 


[65] J. Darlington and M. J. Reeve. Alice and fhe parallel evaluafion of 
logic programs. In Computer Architecture Symposium, Sfockholm, 


June 1983. 26 


[66] Pierre-Gharles David and Thomas Ledoux. An aspecf-orienfed ap¬ 
proach for developing self-adapfive fracfal componenfs. In Welf 
Lowe and Mario Siidholf, edifors. Proceedings of the 5th Inti Sympo¬ 
sium Software on Composition, volume 4089 of Lecture Notes in Com¬ 
puter Science, pages 82-97, Vienna, Ausfria, March 2006. Springer. 

m\ 

[67] Alexandre Denis, Ghrisfian Perez, Thierry Priol, and Andre Ribes. 
Bringing high performance fo fhe corba componenf model. In 
SIAM Conference on Parallel Processing for Scientific Computing, 
February 2004. |6 63 


[68] T. Elrad, R. E. Eilman, and A. Bader. Aspecf-orienfed program¬ 


ming. Communications of the ACM, 44(10), Ocf. 2001. 62 


[69] Joao Eemando Eerreira, Joao Luis Sobral, and Alberto Jose Proen^a. 
Jaskel: A java skeleton-based framework for sfrucfured clusfer and 


grid computing. In CCGRID, pages 301-304. IEEE, 2006. 68 


[70] Marc Eleury. The Official JBoss Development and Administration 
Guide. Pearson Education, 2002. 

[71] Ivan Eufo. Prolog wifh communicating processes: from f-prolog fo 
csr-prolog. In ICLP'93: Proceedings of the tenth international confer¬ 
ence on logic programming on Logic programming, pages 3-17, Gam- 


bridge, MA, USA, 1993. MIT Press. 26 


163 












[72] Vladimir Getov, Gregor von Laszewski, Michael Philippsen, and 
Ian Foster. Multiparadigm communications in java for grid com¬ 
puting. Commun. ACM, 44(10):118-125, 2001. |6 63 


[73] S. Gorlatch and J. Diinnweber. From grid middleware to grid ap¬ 
plications: Bridging the gap with HOGs. In V. Getov, D. Laforenza, 
and A. Reinefeld, editors. Future Generation Grids, GoreGRID se¬ 


ries. Springer, November 2005. 59 130 


[74] Survey on grid 

http: / / wiki.cogkit.org / index.php / 


[TToIlm] 


workflows, 2007. 

Survey _on_Grid .Workflows. 


[75] Andrew S. Grimshaw. The menfaf compufafion model dafa-driven 
support for objecf-orienfed parallel processing. Technical reporf. 


Depf. Comp. Science, Univ. Virginia, 28 1993. 32 


[76] Andrew S. Grimshaw and Jane W. S. Liu. Menfaf: An objecf- 
oriented macro dafa flow sysfem. In OOPSLA '87: Conference pro¬ 
ceedings on Object-oriented programming systems, languages and appli¬ 


cations, pages 35-47, New York, NY, USA, 1987. ACM. 30 


[77] Nexf Generafion GRIDs Experf Group. NGG3, Future for European 
Grids: GRIDs and Service Oriented Knowledge Utilities. Vision and Re¬ 
search Directions 2010 and Beyond. Nexf Generafion GRIDs Experf 


Group, January 2006. 124 


[78] Bruno Harbulof and John R. Curd. Using aspecfj fo separafe con¬ 
cerns in parallel scienfific java code. In Proceedings of the 3rd Inter¬ 
national Conference on Aspect-Oriented Software Development, pages 
122-131, Lancasfer, UK, March 2004. |96] 


[79] Rod Johnson, Juergen Hoeller, Alef Arendsen, Thomas Risberg, 
and Dmifriy Kopylenko. Professional Java Development with the 
Spring Framework. Wrox Press Lfd., Birmingham, UK, UK, 2005. 

mnn 


164 












[80] Wesley M. Johnston, J. R. Paul Hanna, and Richard J. Millar. Ad¬ 
vances in dataflow programming languages. ACM Computing Sur¬ 


veys, 36(l):l-34,2004. 27 


[81] Paul H. Kelly. Functional Programming for Loosely-Coupled Multipro¬ 


cessors. MIT Press, Cambridge, MA, USA, 1989. 24 


[82] K. Kennedy, M. Mazina, J. Mellor-Crummey, K. Cooper, L. Torc- 
zon, F. Berman, A. Chien, H. Bail, O. Sievert, D. Angulo, I. Foster, 
D. Garmon, L. Johnsson, C. Kesselman, R. Aydt, D. Reed, J. Don- 
garra, S. Vadhiyar, and R. Wolski. Toward a framework for prepar¬ 
ing and execufing adaptive Grid programs. In Proceedings of the 
NSF Next Generation Systems Program Workshop (IPDPS 2002), 2002. 
[123] 


[83] J. O. Kephart and D. M. Chess. The vision of autonomic computing. 


IEEE Computer, 36(l):41-50, 2003. 124 127 


[84] Gregor Kiczales, Erik Hilsdale, Jim Hugunin, Mik Kersfen, Jeffrey 
Palm, and William Griswold. Getting sfarfed wifh aspectj. Com¬ 


munications of the ACM, 44(10):59-65, 2001. 112 


[85] Gregor Kiczales, John Lamping, Anurag Mendhekar, Chris Maeda, 
Crisfina Lopes, Jean-Marc Loingfier, and John Irwin. Aspecf- 
orienfed programming. In Proceedings of the European Conference 
on Object-Oriented Programming, volume 1241, page 220242, 1997. 
HD 


[86] A. Krall. Eficienfjavavm jusf-in-fimecompilafion. In Jean-Luc Gau- 
diof, edifor. Proceedings of the International Conference on Parallel Ar¬ 
chitectures and Compilation Techniques, pages 205-212, Paris, 1998. 
|92| 


[87] H. Kuchen. Optimizing sequences of skeleton calls. In D. Bafory, 
C. Gonsel, C. Lengauer, and M. Odersky, edifors, Domain-Specific 
Program Generation, number 3016 in Lecfure Nofes in Gompufer 


Science, pages 254-273. Springer Verlag, 2004. 66 67 


165 













[88] Herbert Kuchen. A skeleton library. In B. Monien and R. Feld¬ 
man, editors. Proceedings of the 8th International Euro-Par Conference, 
volume 2400 of Lecture Notes in Computer Science, pages 620-629, 


Paderbom, Germany, August 2002. Springer. 48 66 67 


[89] Herbert Kuchen. The muesli home page, 2006. http: / /www. wi , 
uni-muenster.de/PI/forschung/Skeletons/ 


[90] Vipin Kumar, Ananth Grama, Anshul Gupta, and George Karypis. 
Introduction to parallel computing: design and analysis of algorithms. 
Benjamin-Cummings Publishing Co., Inc., Redwood City, CA, 
USA, 1994. |5^ 


[91] T. Lindholm and F. Yellin. The Java Virtual Machine Specification. 


Addison-Wesley, 1999. 92 


[92] S. McDonald, D. Szafron, J. Schaeffer, and S. Bromling. Generaf- 
ing parallel program frameworks from parallel design paffems. In 
A. Bode, T. Ludwing, W. Karl, and R. Wismiiller, editors. Proceed¬ 
ings of the 6th International Euro-Par Conference, LNCS, No. 1900, 


pages 95-105. Springer Verlag, August/September 2000. 68 


[93] Manish Parashar, Hua Liu, Zhen Li, Vincent Matossian, Cristina 
Schmidt, Guangsen Zhang, and Salim Hariri. AutoMate: Enabling 
autonomic applications on the Grid. Cluster Computing, 9(2):161- 
174, 2006. [T^ 

[94] S. Pelagatti. Structured Development of Parallel Programs. Taylor and 
Francis, 1998. 1^ 


[95] Luis Moniz Pereira, Luis Monteiro, Jose Cunha, and Joaquim N 
Aparicio. Delta prolog: a distributed backtracking extension with 
events. In Proceedings on Third international conference on logic pro¬ 
gramming, pages 69-83, New York, NY, USA, 1986. Springer Verlag. 
|26| 


[96] M. Poldner and H. Kuchen. Scalable farms. In Proceedings of the 
Parallel Computing (ParCo), 2005. Malaga (posf-proceedings, fo ap- 


166 












pear, available at http : / /danae . uni-muenster . de/ lehre/ 
kuchen/ papersi. html. |70| 


[97] R. Rouvoy, N. Pessemier, R. Pawlak, and P Merle. Using attribute- 
oriented programming to leverage fractal-based developments. In 


Proceedings of the 5th Fractal workshop at ECOOP 2006, July 2006. 61 


[98] Romain Rouvoy and Philippe Merle. Leveraging component- 
oriented programming with attribute-oriented programming. In 
Proceedings of the 11th International ECOOP Workshop on Component- 
Oriented Programming, volume 2006-11, Nantes, France, July 2006. 


Karlsruhe University. 94 


[99] S. M. Sadjadi and P. K. McKinley. Transparent self-optimization 
in existing CORBA applications. In Proceedings of the 1st Interna¬ 
tional Conference on Autonomic Computing (ICAC'04), pages 88-95, 


Washington, DC, USA, 2004. IEEE. 133 


[100] J. Serot. Tagged-token data-flow for skeletons. Parallel Processing 
Letters, 11(4), 2001. 2001. ^ 

[101] Jocel 5 m Serof and Dominique Ginhac. Skelefons for parallel image 
processing: an overview of fhe skipper projecf. Parallel Computing, 


28(12):1685-1708,2002. 46 5^^ 


[102] E. Shapiro, edifor. Concurrent prolog: collected papers. MIT Press, 


Cambridge, MA, USA, 1987. 26 


[103] J. Silc, B. Robic, and T. Ungerer. As 5 mchrony in parallel computing: 
Prom dafaflow to mulfifhreading, 1998. 


[104] Dezso Sima, Terence Pounfain, and Peter Karsuk. Advanced Com¬ 
puter Architectures: A Design Space Approach. Addison-Wesley, 1997. 

M 


[105] B.C. Smifh and J. des Rivieres. 


XEROX, Palo Alfo, June 1984. 93 


Interim 3-LISP Reference Manual. 


167 









[106] J. Sobral. Incrementally developing parallel applications with as¬ 
pect). In Proceedings of the 20th IEEE International Parallel and Dis¬ 
tributed Processing Symposium. IEEE Press, 4 2006. 

[107] Joao L. Sobral, Miguel P. Monteiro, and Carlos A. Cunha. Aspect- 
oriented support for modular parallel computing. In Proceed¬ 
ings of the 5th Workshop on Aspects, Components, and Patterns for 
Infrastructure Software, pages 37-41, Bonn, Germany, 2006. Pub¬ 
lished as University of Virginia Compufer Science Technical Reporf 
CS-2006-01. M 

[108] OASIS feam. Proacfive home page, 2006. 

http://www-sop.inria.fr/oasis/proactive/. [5^ 

[109] P. Tefi. Lifhium: a java skeleton environmenf in italian. Master's 
fhesis, Depf. Compufer Science, Universify of Pisa, Ocfober 2001. 
IZI] 

[110] Prank Yellin Tim Lindholm. The Java Virtual Machine Specification. 
Sun Microsystems Press, second edition edifion, 2004. 101| 


[111] E. Truyen, B. Jorgensen, W. Joosen, and P. Verbaefen. On infer- 
acfion refinemenf in middleware. In J. Bosch, C. Sz 5 rperski, and 
W. Week, editors. Proceedings of the 5th International Workshop on 


Component-Oriented Programming, pages 56-62, 2001. 133 


[112] Kazunori Ueda. Guarded Horn clauses. D.eng. fhesis, Universify of 


Tokyo, Tokyo, Japan, March 1986. 26 


[113] Marco Vanneschi. The programming model of ASSIST, an envi¬ 
ronmenf for parallel and disfribufed porfable applications. Parallel 


Computing, 28(12):1709-1732, December 2002. 48 125 


[114] Hiroshi Wada and Junichi Suzuki. Modeling fumpike fronfend 
sysfem: A model-driven developmenf framework leveraging uml 
mefamodeling and affribufe-orienfed programming. In MoDELS, 


pages 584-600,2005. 94 


168 












[115] Paul G. Whiting and Robert S. V. Pascoe. A history of data-flow 
languages. IEEE Annals of the History of Computing, 16(4):38-59, 
1994. HZl 


169 


170 






SOME RIGHTS RESERVED 

0 ®(§) 


Unless otherwise expressly stated, all original material of whatever 
nature created by Patrizio Dazzi and included in this thesis, is li¬ 
censed under the Creative Commons Attribution Non-commercial 
Share Alike 2.5 Italy License 

Check creativecommons. 0 rg/licenses/by-nc-sa/ 2 . 5 /it/ for the legal 
code of the full license. 


Ask the author about other uses. 




